2026-02-25T08:03:25.175 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-25T08:03:25.190 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T08:03:25.268 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69078 branch: wip-logstoremanager description: crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} thrashers/{default} thrashosds-health workloads/snaps-few-objects-balanced} email: null first_in_suite: false flavor: debug job_id: '69078' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-logstoremanager ceph: conf: global: enable experimental unrecoverable data corrupting features: crimson ms client mode: crc ms cluster mode: crc ms mon client mode: crc ms mon cluster mode: crc ms mon service mode: crc ms service mode: crc osd_pool_default_min_size: 2 osd_pool_default_size: 3 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 osd pool default crimson: true paxos service trim min: 10 osd: crimson cpu num: 4 crimson osd obc lru size: 10 debug ms: 20 debug osd: 20 debug seastore: 20 debug seastore backref: 20 debug seastore cache: 20 debug seastore cleaner: 20 debug seastore device: 20 debug seastore epm: 20 debug seastore fixedkv tree: 20 debug seastore journal: 20 debug seastore lba: 20 debug seastore odata: 20 debug seastore omap: 20 debug seastore onode: 20 debug seastore t: 20 debug seastore tm: 20 osd blocked scrub grace period: 3600 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd max markdown count: 1000 osd objectstore: seastore osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 osd target transaction size: 5 seastore cachepin size pershard: 64M seastore main device type: RANDOM_BLOCK_SSD seastore max concurrent transactions: 8 crimson_compat: true flavor: debug log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(POOL_APP_NOT_ENABLED\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - SLOW_OPS - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub - Degraded data redundancy - is down - osds down - pg .*? is .*?degraded.*?, acting - pg .*? is stuck - pg degraded - PG_BACKFILL_FULL - Low space hindering backfill .*? backfill_toofull - OSD_ROOT_DOWN - pgs degraded - pgs undersized - is active.*backfill_toofull.* sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: debug sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 thrashosds: min_in: 4 workunit: branch: main env: CRIMSON_COMPAT: '1' sha1: c32469671c6b0222f786dd36c34eb702f780a11d owner: scheduled_matan@soko04.front.sepia.ceph.com priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - osd.0 - osd.1 - client.0 - node-exporter.a - - mgr.x - osd.2 - osd.3 - client.1 - prometheus.a - node-exporter.b seed: 869 sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 sleep_before_teardown: 0 suite: crimson-rados suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph.git suite_sha1: c32469671c6b0222f786dd36c34eb702f780a11d targets: trial169.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOxFCEh2riEJZuVhsl0d7Lfw0dtKL2gecErtNAwJLjCJMKGYEp+znTw+NUtgwFx3fzSCf/D+SrhoKzz6wLSPJAM= trial183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBhkL8/E3SLJ4IZT5lytGCGhBDsJdcorAIMTMxmCMPZ1j8Mnhva1PeYwM7RBGbB9XOqhi8yyi/7PVrz1+3G5FEo= tasks: - install: extra_packages: deb: - ceph-osd-crimson - ceph-osd-crimson-dbg rpm: - ceph-osd-crimson - ceph-osd-crimson-debuginfo - pexec: all: - sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson - install: null - ceph: conf: osd: debug monc: 20 - thrashosds: ceph_objectstore_tool: false chance_inject_pause_short: 0 chance_reset_purged_snaps_last: 0 chance_thrash_cluster_full: 0 chance_thrash_pg_upmap: 0 chance_trim_stale_osdmaps: 0 dump_ops_enable: false min_in: 3 noscrub_toggle_delay: 0 reweight_osd: 0 sighup_delay: 0 thrash_primary_affinity: false timeout: 2400 - rados: balance_reads: true clients: - client.0 objects: 50 op_weights: copy_from: 0 delete: 50 read: 100 rollback: 0 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-25_07:55:35 tube: trial user: matan verbose: false worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-25T08:03:25.268 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa; will attempt to use it 2026-02-25T08:03:25.269 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks 2026-02-25T08:03:25.269 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-25T08:03:25.270 INFO:teuthology.task.internal:Checking packages... 2026-02-25T08:03:25.270 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'debug' and ceph hash '6aeb6975bc9a4790a399c9cc70dba9635d2a7819' 2026-02-25T08:03:25.270 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-25T08:03:25.271 INFO:teuthology.packaging:ref: None 2026-02-25T08:03:25.271 INFO:teuthology.packaging:tag: None 2026-02-25T08:03:25.271 INFO:teuthology.packaging:branch: wip-logstoremanager 2026-02-25T08:03:25.271 INFO:teuthology.packaging:sha1: 6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:03:25.271 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&ref=wip-logstoremanager 2026-02-25T08:03:25.403 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5338.g6aeb6975 2026-02-25T08:03:25.404 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-25T08:03:25.415 INFO:teuthology.task.internal:no buildpackages task found 2026-02-25T08:03:25.415 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-25T08:03:25.429 INFO:teuthology.task.internal:Saving configuration 2026-02-25T08:03:25.443 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-25T08:03:25.456 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-25T08:03:25.534 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial169.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69078', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 07:57:48.572902', 'locked_by': 'scheduled_matan@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOxFCEh2riEJZuVhsl0d7Lfw0dtKL2gecErtNAwJLjCJMKGYEp+znTw+NUtgwFx3fzSCf/D+SrhoKzz6wLSPJAM='} 2026-02-25T08:03:25.590 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial183.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69078', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 07:57:48.573471', 'locked_by': 'scheduled_matan@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBhkL8/E3SLJ4IZT5lytGCGhBDsJdcorAIMTMxmCMPZ1j8Mnhva1PeYwM7RBGbB9XOqhi8yyi/7PVrz1+3G5FEo='} 2026-02-25T08:03:25.590 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-25T08:03:25.602 INFO:teuthology.task.internal:roles: ubuntu@trial169.front.sepia.ceph.com - ['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'] 2026-02-25T08:03:25.602 INFO:teuthology.task.internal:roles: ubuntu@trial183.front.sepia.ceph.com - ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b'] 2026-02-25T08:03:25.603 INFO:teuthology.run_tasks:Running task console_log... 2026-02-25T08:03:25.736 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x738354d32ac0>, signals=[15]) 2026-02-25T08:03:25.736 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-25T08:03:25.748 INFO:teuthology.task.internal:Opening connections... 2026-02-25T08:03:25.748 DEBUG:teuthology.task.internal:connecting to ubuntu@trial169.front.sepia.ceph.com 2026-02-25T08:03:25.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:03:25.812 DEBUG:teuthology.task.internal:connecting to ubuntu@trial183.front.sepia.ceph.com 2026-02-25T08:03:25.813 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:03:25.876 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-25T08:03:25.894 DEBUG:teuthology.orchestra.run.trial169:> uname -m 2026-02-25T08:03:25.912 INFO:teuthology.orchestra.run.trial169.stdout:x86_64 2026-02-25T08:03:25.912 DEBUG:teuthology.orchestra.run.trial169:> cat /etc/os-release 2026-02-25T08:03:25.967 INFO:teuthology.orchestra.run.trial169.stdout:NAME="CentOS Stream" 2026-02-25T08:03:25.968 INFO:teuthology.orchestra.run.trial169.stdout:VERSION="9" 2026-02-25T08:03:25.968 INFO:teuthology.orchestra.run.trial169.stdout:ID="centos" 2026-02-25T08:03:25.968 INFO:teuthology.orchestra.run.trial169.stdout:ID_LIKE="rhel fedora" 2026-02-25T08:03:25.968 INFO:teuthology.orchestra.run.trial169.stdout:VERSION_ID="9" 2026-02-25T08:03:25.968 INFO:teuthology.orchestra.run.trial169.stdout:PLATFORM_ID="platform:el9" 2026-02-25T08:03:25.968 INFO:teuthology.orchestra.run.trial169.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-25T08:03:25.968 INFO:teuthology.orchestra.run.trial169.stdout:ANSI_COLOR="0;31" 2026-02-25T08:03:25.968 INFO:teuthology.orchestra.run.trial169.stdout:LOGO="fedora-logo-icon" 2026-02-25T08:03:25.968 INFO:teuthology.orchestra.run.trial169.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-25T08:03:25.968 INFO:teuthology.orchestra.run.trial169.stdout:HOME_URL="https://centos.org/" 2026-02-25T08:03:25.968 INFO:teuthology.orchestra.run.trial169.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-25T08:03:25.968 INFO:teuthology.orchestra.run.trial169.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-25T08:03:25.968 INFO:teuthology.orchestra.run.trial169.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-25T08:03:25.969 INFO:teuthology.lock.ops:Updating trial169.front.sepia.ceph.com on lock server 2026-02-25T08:03:26.065 DEBUG:teuthology.orchestra.run.trial183:> uname -m 2026-02-25T08:03:26.080 INFO:teuthology.orchestra.run.trial183.stdout:x86_64 2026-02-25T08:03:26.080 DEBUG:teuthology.orchestra.run.trial183:> cat /etc/os-release 2026-02-25T08:03:26.134 INFO:teuthology.orchestra.run.trial183.stdout:NAME="CentOS Stream" 2026-02-25T08:03:26.134 INFO:teuthology.orchestra.run.trial183.stdout:VERSION="9" 2026-02-25T08:03:26.134 INFO:teuthology.orchestra.run.trial183.stdout:ID="centos" 2026-02-25T08:03:26.134 INFO:teuthology.orchestra.run.trial183.stdout:ID_LIKE="rhel fedora" 2026-02-25T08:03:26.134 INFO:teuthology.orchestra.run.trial183.stdout:VERSION_ID="9" 2026-02-25T08:03:26.134 INFO:teuthology.orchestra.run.trial183.stdout:PLATFORM_ID="platform:el9" 2026-02-25T08:03:26.134 INFO:teuthology.orchestra.run.trial183.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-25T08:03:26.134 INFO:teuthology.orchestra.run.trial183.stdout:ANSI_COLOR="0;31" 2026-02-25T08:03:26.134 INFO:teuthology.orchestra.run.trial183.stdout:LOGO="fedora-logo-icon" 2026-02-25T08:03:26.134 INFO:teuthology.orchestra.run.trial183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-25T08:03:26.134 INFO:teuthology.orchestra.run.trial183.stdout:HOME_URL="https://centos.org/" 2026-02-25T08:03:26.135 INFO:teuthology.orchestra.run.trial183.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-25T08:03:26.135 INFO:teuthology.orchestra.run.trial183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-25T08:03:26.135 INFO:teuthology.orchestra.run.trial183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-25T08:03:26.135 INFO:teuthology.lock.ops:Updating trial183.front.sepia.ceph.com on lock server 2026-02-25T08:03:26.205 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-25T08:03:26.218 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-25T08:03:26.230 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-25T08:03:26.230 DEBUG:teuthology.orchestra.run.trial169:> test '!' -e /home/ubuntu/cephtest 2026-02-25T08:03:26.232 DEBUG:teuthology.orchestra.run.trial183:> test '!' -e /home/ubuntu/cephtest 2026-02-25T08:03:26.248 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-25T08:03:26.258 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-25T08:03:26.258 DEBUG:teuthology.orchestra.run.trial169:> test -z $(ls -A /var/lib/ceph) 2026-02-25T08:03:26.290 DEBUG:teuthology.orchestra.run.trial183:> test -z $(ls -A /var/lib/ceph) 2026-02-25T08:03:26.306 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-25T08:03:26.461 INFO:teuthology.run_tasks:Running task kernel... 2026-02-25T08:03:26.480 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-25T08:03:26.480 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'mgr.x': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-25T08:03:26.480 DEBUG:teuthology.orchestra.run.trial169:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T08:03:26.480 DEBUG:teuthology.orchestra.run.trial183:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T08:03:26.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:03:26.495 DEBUG:teuthology.orchestra.run.trial183:> uname -r 2026-02-25T08:03:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:03:26.497 DEBUG:teuthology.orchestra.run.trial169:> uname -r 2026-02-25T08:03:26.550 INFO:teuthology.orchestra.run.trial183.stdout:5.14.0-681.el9.x86_64 2026-02-25T08:03:26.550 INFO:teuthology.task.kernel:Running kernel on trial183: 5.14.0-681.el9.x86_64 2026-02-25T08:03:26.550 DEBUG:teuthology.orchestra.run.trial183:> sudo yum install -y kernel 2026-02-25T08:03:26.553 INFO:teuthology.orchestra.run.trial169.stdout:5.14.0-681.el9.x86_64 2026-02-25T08:03:26.554 INFO:teuthology.task.kernel:Running kernel on trial169: 5.14.0-681.el9.x86_64 2026-02-25T08:03:26.554 DEBUG:teuthology.orchestra.run.trial169:> sudo yum install -y kernel 2026-02-25T08:03:30.228 INFO:teuthology.orchestra.run.trial169.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.9 MB 00:03 2026-02-25T08:03:30.556 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - BaseOS 2.5 MB/s | 8.9 MB 00:03 2026-02-25T08:03:33.101 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - AppStream 13 MB/s | 27 MB 00:02 2026-02-25T08:03:33.333 INFO:teuthology.orchestra.run.trial169.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-25T08:03:36.436 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 8.0 MB 00:00 2026-02-25T08:03:36.823 INFO:teuthology.orchestra.run.trial169.stdout:CentOS Stream 9 - CRB 7.0 MB/s | 8.0 MB 00:01 2026-02-25T08:03:38.099 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 20 kB 00:00 2026-02-25T08:03:38.349 INFO:teuthology.orchestra.run.trial183.stdout:Extra Packages for Enterprise Linux 109 MB/s | 20 MB 00:00 2026-02-25T08:03:38.610 INFO:teuthology.orchestra.run.trial169.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 20 kB 00:01 2026-02-25T08:03:39.333 INFO:teuthology.orchestra.run.trial169.stdout:Extra Packages for Enterprise Linux 31 MB/s | 20 MB 00:00 2026-02-25T08:03:41.573 INFO:teuthology.orchestra.run.trial183.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-25T08:03:42.533 INFO:teuthology.orchestra.run.trial169.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-25T08:03:42.579 INFO:teuthology.orchestra.run.trial183.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-25T08:03:42.579 INFO:teuthology.orchestra.run.trial183.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-25T08:03:42.599 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:03:42.600 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:03:42.600 INFO:teuthology.orchestra.run.trial183.stdout: Package Architecture Version Repository Size 2026-02-25T08:03:42.600 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:03:42.600 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-25T08:03:42.600 INFO:teuthology.orchestra.run.trial183.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-25T08:03:42.600 INFO:teuthology.orchestra.run.trial183.stdout:Installing dependencies: 2026-02-25T08:03:42.601 INFO:teuthology.orchestra.run.trial183.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-25T08:03:42.601 INFO:teuthology.orchestra.run.trial183.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-25T08:03:42.601 INFO:teuthology.orchestra.run.trial183.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-25T08:03:42.601 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:03:42.601 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-25T08:03:42.601 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:03:42.601 INFO:teuthology.orchestra.run.trial183.stdout:Install 4 Packages 2026-02-25T08:03:42.601 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:03:42.601 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 89 M 2026-02-25T08:03:42.601 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 135 M 2026-02-25T08:03:42.601 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-25T08:03:43.544 INFO:teuthology.orchestra.run.trial169.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-25T08:03:43.545 INFO:teuthology.orchestra.run.trial169.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-25T08:03:43.564 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:03:43.565 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:03:43.565 INFO:teuthology.orchestra.run.trial169.stdout: Package Architecture Version Repository Size 2026-02-25T08:03:43.565 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:03:43.565 INFO:teuthology.orchestra.run.trial169.stdout:Installing: 2026-02-25T08:03:43.565 INFO:teuthology.orchestra.run.trial169.stdout: kernel x86_64 5.14.0-686.el9 baseos 901 k 2026-02-25T08:03:43.565 INFO:teuthology.orchestra.run.trial169.stdout:Installing dependencies: 2026-02-25T08:03:43.565 INFO:teuthology.orchestra.run.trial169.stdout: kernel-core x86_64 5.14.0-686.el9 baseos 17 M 2026-02-25T08:03:43.565 INFO:teuthology.orchestra.run.trial169.stdout: kernel-modules x86_64 5.14.0-686.el9 baseos 40 M 2026-02-25T08:03:43.565 INFO:teuthology.orchestra.run.trial169.stdout: kernel-modules-core x86_64 5.14.0-686.el9 baseos 31 M 2026-02-25T08:03:43.565 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:03:43.565 INFO:teuthology.orchestra.run.trial169.stdout:Transaction Summary 2026-02-25T08:03:43.565 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:03:43.565 INFO:teuthology.orchestra.run.trial169.stdout:Install 4 Packages 2026-02-25T08:03:43.565 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:03:43.566 INFO:teuthology.orchestra.run.trial169.stdout:Total download size: 89 M 2026-02-25T08:03:43.566 INFO:teuthology.orchestra.run.trial169.stdout:Installed size: 135 M 2026-02-25T08:03:43.566 INFO:teuthology.orchestra.run.trial169.stdout:Downloading Packages: 2026-02-25T08:03:44.063 INFO:teuthology.orchestra.run.trial183.stdout:(1/4): kernel-core-5.14.0-686.el9.x86_64.rpm 12 MB/s | 17 MB 00:01 2026-02-25T08:03:44.537 INFO:teuthology.orchestra.run.trial169.stdout:(1/4): kernel-5.14.0-686.el9.x86_64.rpm 1.0 MB/s | 901 kB 00:00 2026-02-25T08:03:44.636 INFO:teuthology.orchestra.run.trial183.stdout:(2/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 20 MB/s | 40 MB 00:01 2026-02-25T08:03:45.073 INFO:teuthology.orchestra.run.trial183.stdout:(3/4): kernel-modules-core-5.14.0-686.el9.x86_6 31 MB/s | 31 MB 00:01 2026-02-25T08:03:45.244 INFO:teuthology.orchestra.run.trial183.stdout:(4/4): kernel-5.14.0-686.el9.x86_64.rpm 349 kB/s | 901 kB 00:02 2026-02-25T08:03:45.245 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:03:45.246 INFO:teuthology.orchestra.run.trial183.stdout:Total 34 MB/s | 89 MB 00:02 2026-02-25T08:03:45.435 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-25T08:03:45.634 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-25T08:03:45.634 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-25T08:03:46.021 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-25T08:03:46.021 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-25T08:03:47.131 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-25T08:03:47.424 INFO:teuthology.orchestra.run.trial183.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:03:47.446 INFO:teuthology.orchestra.run.trial183.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:03:47.710 INFO:teuthology.orchestra.run.trial169.stdout:(2/4): kernel-modules-core-5.14.0-686.el9.x86_6 9.8 MB/s | 31 MB 00:03 2026-02-25T08:03:50.039 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:03:50.072 INFO:teuthology.orchestra.run.trial183.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:03:53.175 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:03:53.182 INFO:teuthology.orchestra.run.trial183.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:03:56.234 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:04.813 INFO:teuthology.orchestra.run.trial169.stdout:(3/4): kernel-core-5.14.0-686.el9.x86_64.rpm 832 kB/s | 17 MB 00:21 2026-02-25T08:04:06.264 INFO:teuthology.orchestra.run.trial169.stdout:(4/4): kernel-modules-5.14.0-686.el9.x86_64.rpm 1.8 MB/s | 40 MB 00:22 2026-02-25T08:04:06.266 INFO:teuthology.orchestra.run.trial169.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:04:06.266 INFO:teuthology.orchestra.run.trial169.stdout:Total 3.9 MB/s | 89 MB 00:22 2026-02-25T08:04:06.458 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction check 2026-02-25T08:04:06.660 INFO:teuthology.orchestra.run.trial169.stdout:Transaction check succeeded. 2026-02-25T08:04:06.660 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction test 2026-02-25T08:04:07.042 INFO:teuthology.orchestra.run.trial169.stdout:Transaction test succeeded. 2026-02-25T08:04:07.042 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction 2026-02-25T08:04:08.152 INFO:teuthology.orchestra.run.trial169.stdout: Preparing : 1/1 2026-02-25T08:04:08.442 INFO:teuthology.orchestra.run.trial169.stdout: Installing : kernel-modules-core-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:04:08.463 INFO:teuthology.orchestra.run.trial169.stdout: Installing : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:04:11.039 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:04:11.072 INFO:teuthology.orchestra.run.trial169.stdout: Installing : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:04:13.696 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:13.719 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:13.905 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:13.905 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:04:13.905 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:04:13.905 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:04:13.962 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:13.962 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:04:13.963 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-25T08:04:13.963 INFO:teuthology.orchestra.run.trial183.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:13.963 INFO:teuthology.orchestra.run.trial183.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-25T08:04:13.963 INFO:teuthology.orchestra.run.trial183.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-25T08:04:13.963 INFO:teuthology.orchestra.run.trial183.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-25T08:04:13.963 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:04:13.963 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:04:14.026 DEBUG:teuthology.orchestra.run.trial183:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:04:14.055 INFO:teuthology.orchestra.run.trial183.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:14.056 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:04:14.056 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-25T08:04:14.056 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T08:04:14.056 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T08:04:14.056 INFO:teuthology.task.kernel:Installing distro kernel on mgr.x... 2026-02-25T08:04:14.056 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial183.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T08:04:14.056 DEBUG:teuthology.orchestra.run.trial183:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:04:14.109 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:04:14.115 INFO:teuthology.orchestra.run.trial169.stdout: Installing : kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:14.121 INFO:teuthology.orchestra.run.trial183.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:14.123 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:04:14.123 DEBUG:teuthology.orchestra.run.trial183:> sudo rpm -qi grub2-tools 2026-02-25T08:04:14.191 INFO:teuthology.orchestra.run.trial183.stdout:Name : grub2-tools 2026-02-25T08:04:14.191 INFO:teuthology.orchestra.run.trial183.stdout:Epoch : 1 2026-02-25T08:04:14.191 INFO:teuthology.orchestra.run.trial183.stdout:Version : 2.06 2026-02-25T08:04:14.191 INFO:teuthology.orchestra.run.trial183.stdout:Release : 120.el9 2026-02-25T08:04:14.191 INFO:teuthology.orchestra.run.trial183.stdout:Architecture: x86_64 2026-02-25T08:04:14.191 INFO:teuthology.orchestra.run.trial183.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-25T08:04:14.192 INFO:teuthology.orchestra.run.trial183.stdout:Group : Unspecified 2026-02-25T08:04:14.192 INFO:teuthology.orchestra.run.trial183.stdout:Size : 9108611 2026-02-25T08:04:14.192 INFO:teuthology.orchestra.run.trial183.stdout:License : GPLv3+ 2026-02-25T08:04:14.192 INFO:teuthology.orchestra.run.trial183.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-25T08:04:14.192 INFO:teuthology.orchestra.run.trial183.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-25T08:04:14.192 INFO:teuthology.orchestra.run.trial183.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-25T08:04:14.192 INFO:teuthology.orchestra.run.trial183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-25T08:04:14.192 INFO:teuthology.orchestra.run.trial183.stdout:Packager : builder@centos.org 2026-02-25T08:04:14.192 INFO:teuthology.orchestra.run.trial183.stdout:Vendor : CentOS 2026-02-25T08:04:14.193 INFO:teuthology.orchestra.run.trial183.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-25T08:04:14.193 INFO:teuthology.orchestra.run.trial183.stdout:Summary : Support tools for GRUB. 2026-02-25T08:04:14.193 INFO:teuthology.orchestra.run.trial183.stdout:Description : 2026-02-25T08:04:14.193 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:04:14.193 INFO:teuthology.orchestra.run.trial183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-25T08:04:14.193 INFO:teuthology.orchestra.run.trial183.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-25T08:04:14.193 INFO:teuthology.orchestra.run.trial183.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-25T08:04:14.193 INFO:teuthology.orchestra.run.trial183.stdout:hardware devices. 2026-02-25T08:04:14.193 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:04:14.193 INFO:teuthology.orchestra.run.trial183.stdout:This subpackage provides tools for support of all platforms. 2026-02-25T08:04:14.195 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-25T08:04:14.195 INFO:teuthology.task.kernel:Updating grub on trial183 to boot 5.14.0-686.el9.x86_64 2026-02-25T08:04:14.196 DEBUG:teuthology.orchestra.run.trial183:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-25T08:04:14.259 INFO:teuthology.orchestra.run.trial183.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-25T08:04:14.261 DEBUG:teuthology.orchestra.run.trial183:> sudo command -v grubby && echo yes || echo no 2026-02-25T08:04:14.325 INFO:teuthology.orchestra.run.trial183.stdout:/sbin/grubby 2026-02-25T08:04:14.326 INFO:teuthology.orchestra.run.trial183.stdout:yes 2026-02-25T08:04:14.327 DEBUG:teuthology.orchestra.run.trial183:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-25T08:04:14.392 INFO:teuthology.orchestra.run.trial183.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-25T08:04:14.393 DEBUG:teuthology.orchestra.run.trial183:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-25T08:04:14.466 DEBUG:teuthology.orchestra.run.trial183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-25T08:04:14.673 INFO:teuthology.orchestra.run.trial183.stderr:Generating grub configuration file ... 2026-02-25T08:04:14.741 INFO:teuthology.orchestra.run.trial183.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-25T08:04:14.747 INFO:teuthology.orchestra.run.trial183.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-25T08:04:14.999 INFO:teuthology.orchestra.run.trial183.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-25T08:04:15.005 INFO:teuthology.orchestra.run.trial183.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-25T08:04:15.039 INFO:teuthology.orchestra.run.trial183.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-25T08:04:15.045 INFO:teuthology.orchestra.run.trial183.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-25T08:04:15.077 INFO:teuthology.orchestra.run.trial183.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-25T08:04:15.083 INFO:teuthology.orchestra.run.trial183.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-25T08:04:15.106 INFO:teuthology.orchestra.run.trial183.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-25T08:04:15.113 INFO:teuthology.orchestra.run.trial183.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-25T08:04:15.198 INFO:teuthology.orchestra.run.trial183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T08:04:15.207 INFO:teuthology.orchestra.run.trial183.stderr:done 2026-02-25T08:04:15.209 DEBUG:teuthology.orchestra.run.trial183:> test -d /sys/firmware/efi 2026-02-25T08:04:15.225 DEBUG:teuthology.orchestra.run.trial183:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-25T08:04:15.290 INFO:teuthology.orchestra.run.trial183.stdout:/boot/efi/EFI/BOOT 2026-02-25T08:04:15.290 INFO:teuthology.orchestra.run.trial183.stdout:/boot/efi/EFI/centos 2026-02-25T08:04:15.292 DEBUG:teuthology.orchestra.run.trial183:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-25T08:04:15.357 INFO:teuthology.orchestra.run.trial183.stdout:no 2026-02-25T08:04:15.357 DEBUG:teuthology.orchestra.run.trial183:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-25T08:04:15.423 INFO:teuthology.orchestra.run.trial183.stdout:yes 2026-02-25T08:04:15.423 DEBUG:teuthology.orchestra.run.trial183:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-25T08:04:15.487 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial183 2026-02-25T08:04:15.487 DEBUG:teuthology.orchestra.run.trial183:> sudo bash -c ' 2026-02-25T08:04:15.487 DEBUG:teuthology.orchestra.run.trial183:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T08:04:15.487 DEBUG:teuthology.orchestra.run.trial183:> ' 2026-02-25T08:04:17.233 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:34.820 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: kernel-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:34.841 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: kernel-modules-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:35.027 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: kernel-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:35.027 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : kernel-5.14.0-686.el9.x86_64 1/4 2026-02-25T08:04:35.027 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : kernel-core-5.14.0-686.el9.x86_64 2/4 2026-02-25T08:04:35.027 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : kernel-modules-5.14.0-686.el9.x86_64 3/4 2026-02-25T08:04:35.083 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : kernel-modules-core-5.14.0-686.el9.x86_64 4/4 2026-02-25T08:04:35.083 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:04:35.084 INFO:teuthology.orchestra.run.trial169.stdout:Installed: 2026-02-25T08:04:35.084 INFO:teuthology.orchestra.run.trial169.stdout: kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:35.084 INFO:teuthology.orchestra.run.trial169.stdout: kernel-core-5.14.0-686.el9.x86_64 2026-02-25T08:04:35.084 INFO:teuthology.orchestra.run.trial169.stdout: kernel-modules-5.14.0-686.el9.x86_64 2026-02-25T08:04:35.084 INFO:teuthology.orchestra.run.trial169.stdout: kernel-modules-core-5.14.0-686.el9.x86_64 2026-02-25T08:04:35.084 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:04:35.084 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:04:35.146 DEBUG:teuthology.orchestra.run.trial169:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:04:35.176 INFO:teuthology.orchestra.run.trial169.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:35.177 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:04:35.177 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-681.el9.x86_64 Expected: 5.14.0-686.el9.x86_64 2026-02-25T08:04:35.177 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T08:04:35.177 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T08:04:35.178 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-25T08:04:35.178 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial169.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T08:04:35.178 DEBUG:teuthology.orchestra.run.trial169:> rpm -q kernel | sort -rV | head -n 1 2026-02-25T08:04:35.242 INFO:teuthology.orchestra.run.trial169.stdout:kernel-5.14.0-686.el9.x86_64 2026-02-25T08:04:35.243 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-686.el9.x86_64 2026-02-25T08:04:35.243 DEBUG:teuthology.orchestra.run.trial169:> sudo rpm -qi grub2-tools 2026-02-25T08:04:35.311 INFO:teuthology.orchestra.run.trial169.stdout:Name : grub2-tools 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:Epoch : 1 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:Version : 2.06 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:Release : 120.el9 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:Architecture: x86_64 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:Install Date: Wed 18 Feb 2026 05:18:42 PM UTC 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:Group : Unspecified 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:Size : 9108611 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:License : GPLv3+ 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:Signature : RSA/SHA256, Thu 04 Dec 2025 09:42:59 AM UTC, Key ID 05b555b38483c65d 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:Source RPM : grub2-2.06-120.el9.src.rpm 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:Build Date : Wed 03 Dec 2025 12:10:38 PM UTC 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:Packager : builder@centos.org 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:Vendor : CentOS 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:URL : http://www.gnu.org/software/grub/ 2026-02-25T08:04:35.312 INFO:teuthology.orchestra.run.trial169.stdout:Summary : Support tools for GRUB. 2026-02-25T08:04:35.313 INFO:teuthology.orchestra.run.trial169.stdout:Description : 2026-02-25T08:04:35.313 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:04:35.313 INFO:teuthology.orchestra.run.trial169.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2026-02-25T08:04:35.313 INFO:teuthology.orchestra.run.trial169.stdout:customizable bootloader with modular architecture. It supports a rich 2026-02-25T08:04:35.313 INFO:teuthology.orchestra.run.trial169.stdout:variety of kernel formats, file systems, computer architectures and 2026-02-25T08:04:35.313 INFO:teuthology.orchestra.run.trial169.stdout:hardware devices. 2026-02-25T08:04:35.313 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:04:35.313 INFO:teuthology.orchestra.run.trial169.stdout:This subpackage provides tools for support of all platforms. 2026-02-25T08:04:35.314 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2026-02-25T08:04:35.314 INFO:teuthology.task.kernel:Updating grub on trial169 to boot 5.14.0-686.el9.x86_64 2026-02-25T08:04:35.314 DEBUG:teuthology.orchestra.run.trial169:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f -print -quit 2026-02-25T08:04:35.376 INFO:teuthology.orchestra.run.trial169.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-681.el9.x86_64.conf 2026-02-25T08:04:35.377 DEBUG:teuthology.orchestra.run.trial169:> sudo command -v grubby && echo yes || echo no 2026-02-25T08:04:35.442 INFO:teuthology.orchestra.run.trial169.stdout:/sbin/grubby 2026-02-25T08:04:35.444 INFO:teuthology.orchestra.run.trial169.stdout:yes 2026-02-25T08:04:35.444 DEBUG:teuthology.orchestra.run.trial169:> sudo find /boot/loader/entries -maxdepth 1 -name "*.conf" -type f | grep -F "5.14.0-686.el9.x86_64" | head -n 1 2026-02-25T08:04:35.508 INFO:teuthology.orchestra.run.trial169.stdout:/boot/loader/entries/b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64.conf 2026-02-25T08:04:35.508 DEBUG:teuthology.orchestra.run.trial169:> sudo grub2-set-default b0373be789df49cdafedf423703d2566-5.14.0-686.el9.x86_64 2026-02-25T08:04:35.583 DEBUG:teuthology.orchestra.run.trial169:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2026-02-25T08:04:35.789 INFO:teuthology.orchestra.run.trial169.stderr:Generating grub configuration file ... 2026-02-25T08:04:35.859 INFO:teuthology.orchestra.run.trial169.stderr:Found linux image: /boot/vmlinuz-5.14.0-686.el9.x86_64 2026-02-25T08:04:35.866 INFO:teuthology.orchestra.run.trial169.stderr:Found initrd image: /boot/initramfs-5.14.0-686.el9.x86_64.img 2026-02-25T08:04:36.123 INFO:teuthology.orchestra.run.trial169.stderr:Found linux image: /boot/vmlinuz-5.14.0-681.el9.x86_64 2026-02-25T08:04:36.130 INFO:teuthology.orchestra.run.trial169.stderr:Found initrd image: /boot/initramfs-5.14.0-681.el9.x86_64.img 2026-02-25T08:04:36.165 INFO:teuthology.orchestra.run.trial169.stderr:Found linux image: /boot/vmlinuz-5.14.0-645.el9.x86_64 2026-02-25T08:04:36.172 INFO:teuthology.orchestra.run.trial169.stderr:Found initrd image: /boot/initramfs-5.14.0-645.el9.x86_64.img 2026-02-25T08:04:36.204 INFO:teuthology.orchestra.run.trial169.stderr:Found linux image: /boot/vmlinuz-0-rescue-b0373be789df49cdafedf423703d2566 2026-02-25T08:04:36.211 INFO:teuthology.orchestra.run.trial169.stderr:Found initrd image: /boot/initramfs-0-rescue-b0373be789df49cdafedf423703d2566.img 2026-02-25T08:04:36.234 INFO:teuthology.orchestra.run.trial169.stderr:Found linux image: /boot/vmlinuz-0-rescue-2734456005d743dca7f3dc77c35a2d0d 2026-02-25T08:04:36.240 INFO:teuthology.orchestra.run.trial169.stderr:Found initrd image: /boot/initramfs-0-rescue-2734456005d743dca7f3dc77c35a2d0d.img 2026-02-25T08:04:36.328 INFO:teuthology.orchestra.run.trial169.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T08:04:36.337 INFO:teuthology.orchestra.run.trial169.stderr:done 2026-02-25T08:04:36.339 DEBUG:teuthology.orchestra.run.trial169:> test -d /sys/firmware/efi 2026-02-25T08:04:36.356 DEBUG:teuthology.orchestra.run.trial169:> sudo find /boot/efi/EFI -mindepth 1 -maxdepth 1 -type d || true 2026-02-25T08:04:36.421 INFO:teuthology.orchestra.run.trial169.stdout:/boot/efi/EFI/BOOT 2026-02-25T08:04:36.421 INFO:teuthology.orchestra.run.trial169.stdout:/boot/efi/EFI/centos 2026-02-25T08:04:36.423 DEBUG:teuthology.orchestra.run.trial169:> sudo test -f /boot/efi/EFI/BOOT/grub.cfg && echo yes || echo no 2026-02-25T08:04:36.487 INFO:teuthology.orchestra.run.trial169.stdout:no 2026-02-25T08:04:36.488 DEBUG:teuthology.orchestra.run.trial169:> sudo test -f /boot/efi/EFI/centos/grub.cfg && echo yes || echo no 2026-02-25T08:04:36.554 INFO:teuthology.orchestra.run.trial169.stdout:yes 2026-02-25T08:04:36.555 DEBUG:teuthology.orchestra.run.trial169:> sudo cp -f /boot/grub2/grub.cfg /boot/efi/EFI/centos/grub.cfg 2026-02-25T08:04:36.617 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial169 2026-02-25T08:04:36.617 DEBUG:teuthology.orchestra.run.trial169:> sudo bash -c ' 2026-02-25T08:04:36.617 DEBUG:teuthology.orchestra.run.trial169:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T08:04:36.617 DEBUG:teuthology.orchestra.run.trial169:> ' 2026-02-25T08:04:45.529 INFO:teuthology.task.kernel:Checking client mgr.x for new kernel version... 2026-02-25T08:04:45.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial183.front.sepia.ceph.com' 2026-02-25T08:04:45.530 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:04:59.840 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.183 2026-02-25T08:05:06.660 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-25T08:05:06.660 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial169.front.sepia.ceph.com' 2026-02-25T08:05:06.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:05:08.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial183.front.sepia.ceph.com' 2026-02-25T08:05:08.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:05:20.960 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.169 2026-02-25T08:05:29.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial169.front.sepia.ceph.com' 2026-02-25T08:05:29.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:05:45.053 DEBUG:teuthology.orchestra.run.trial183:> true 2026-02-25T08:05:45.230 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial183.front.sepia.ceph.com' 2026-02-25T08:05:45.230 INFO:teuthology.task.kernel:Checking kernel version of mgr.x, want "5.14.0-686.el9.x86_64"... 2026-02-25T08:05:45.231 DEBUG:teuthology.orchestra.run.trial183:> uname -r 2026-02-25T08:05:45.286 INFO:teuthology.orchestra.run.trial183.stdout:5.14.0-686.el9.x86_64 2026-02-25T08:05:45.286 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-25T08:05:45.286 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T08:05:45.286 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-25T08:05:46.287 INFO:teuthology.task.kernel:Enabling kdb on mgr.x... 2026-02-25T08:05:46.287 DEBUG:teuthology.orchestra.run.trial183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T08:05:46.317 INFO:teuthology.orchestra.run.trial183.stdout:ttyS1 2026-02-25T08:05:46.336 DEBUG:teuthology.parallel:result is None 2026-02-25T08:06:06.050 DEBUG:teuthology.orchestra.run.trial169:> true 2026-02-25T08:06:06.213 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial169.front.sepia.ceph.com' 2026-02-25T08:06:06.213 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-686.el9.x86_64"... 2026-02-25T08:06:06.213 DEBUG:teuthology.orchestra.run.trial169:> uname -r 2026-02-25T08:06:06.269 INFO:teuthology.orchestra.run.trial169.stdout:5.14.0-686.el9.x86_64 2026-02-25T08:06:06.269 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-686.el9.x86_64 vs 5.14.0-686.el9.x86_64 2026-02-25T08:06:06.270 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T08:06:06.270 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2026-02-25T08:06:07.270 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-25T08:06:07.271 DEBUG:teuthology.orchestra.run.trial169:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T08:06:07.301 INFO:teuthology.orchestra.run.trial169.stdout:ttyS1 2026-02-25T08:06:07.328 DEBUG:teuthology.parallel:result is None 2026-02-25T08:06:07.328 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-25T08:06:07.338 INFO:teuthology.task.internal:Creating test directory... 2026-02-25T08:06:07.339 DEBUG:teuthology.orchestra.run.trial169:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T08:06:07.370 DEBUG:teuthology.orchestra.run.trial183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T08:06:07.389 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-25T08:06:07.398 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-25T08:06:07.409 INFO:teuthology.task.internal:Creating archive directory... 2026-02-25T08:06:07.409 DEBUG:teuthology.orchestra.run.trial169:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T08:06:07.426 DEBUG:teuthology.orchestra.run.trial183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T08:06:07.448 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-25T08:06:07.459 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-25T08:06:07.459 DEBUG:teuthology.orchestra.run.trial169:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T08:06:07.482 DEBUG:teuthology.orchestra.run.trial183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T08:06:07.504 INFO:teuthology.orchestra.run.trial169.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:06:07.508 INFO:teuthology.orchestra.run.trial183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:06:07.512 INFO:teuthology.orchestra.run.trial169.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:06:07.517 INFO:teuthology.orchestra.run.trial183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T08:06:07.518 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-25T08:06:07.532 INFO:teuthology.task.internal:Configuring sudo... 2026-02-25T08:06:07.533 DEBUG:teuthology.orchestra.run.trial169:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T08:06:07.555 DEBUG:teuthology.orchestra.run.trial183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T08:06:07.582 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-25T08:06:07.598 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-25T08:06:07.599 DEBUG:teuthology.orchestra.run.trial169:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T08:06:07.619 DEBUG:teuthology.orchestra.run.trial183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T08:06:07.638 DEBUG:teuthology.orchestra.run.trial169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:06:07.754 DEBUG:teuthology.orchestra.run.trial169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:06:07.780 DEBUG:teuthology.orchestra.run.trial169:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:06:07.903 DEBUG:teuthology.orchestra.run.trial169:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:06:07.929 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:06:07.929 DEBUG:teuthology.orchestra.run.trial169:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T08:06:07.993 DEBUG:teuthology.orchestra.run.trial183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:06:08.075 DEBUG:teuthology.orchestra.run.trial183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:06:08.101 DEBUG:teuthology.orchestra.run.trial183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:06:08.216 DEBUG:teuthology.orchestra.run.trial183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:06:08.239 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:06:08.239 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T08:06:08.303 DEBUG:teuthology.orchestra.run.trial169:> sudo service rsyslog restart 2026-02-25T08:06:08.305 DEBUG:teuthology.orchestra.run.trial183:> sudo service rsyslog restart 2026-02-25T08:06:08.333 INFO:teuthology.orchestra.run.trial169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:06:08.371 INFO:teuthology.orchestra.run.trial183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:06:08.893 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-25T08:06:08.904 INFO:teuthology.task.internal:Starting timer... 2026-02-25T08:06:08.904 INFO:teuthology.run_tasks:Running task pcp... 2026-02-25T08:06:08.923 INFO:teuthology.run_tasks:Running task selinux... 2026-02-25T08:06:08.935 DEBUG:teuthology.orchestra.run.trial169:> sudo service auditd rotate 2026-02-25T08:06:08.965 INFO:teuthology.orchestra.run.trial169.stdout:Rotating logs: 2026-02-25T08:06:08.966 DEBUG:teuthology.orchestra.run.trial183:> sudo service auditd rotate 2026-02-25T08:06:08.995 INFO:teuthology.orchestra.run.trial183.stdout:Rotating logs: 2026-02-25T08:06:08.997 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-25T08:06:08.997 DEBUG:teuthology.orchestra.run.trial169:> /usr/sbin/getenforce 2026-02-25T08:06:09.022 INFO:teuthology.orchestra.run.trial169.stdout:Permissive 2026-02-25T08:06:09.022 DEBUG:teuthology.orchestra.run.trial183:> /usr/sbin/getenforce 2026-02-25T08:06:09.053 INFO:teuthology.orchestra.run.trial183.stdout:Permissive 2026-02-25T08:06:09.053 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial169.front.sepia.ceph.com': 'permissive', 'ubuntu@trial183.front.sepia.ceph.com': 'permissive'} 2026-02-25T08:06:09.053 DEBUG:teuthology.orchestra.run.trial169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:06:09.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:06:09.086 DEBUG:teuthology.orchestra.run.trial183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:06:09.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:06:09.118 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-25T08:06:09.118 DEBUG:teuthology.orchestra.run.trial169:> sudo /usr/sbin/setenforce permissive 2026-02-25T08:06:09.150 DEBUG:teuthology.orchestra.run.trial183:> sudo /usr/sbin/setenforce permissive 2026-02-25T08:06:09.182 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-25T08:06:09.195 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-25T08:06:09.201 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-25T08:06:09.202 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial169.front.sepia.ceph.com,trial183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-25T08:08:49.080 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial169.front.sepia.ceph.com'), Remote(name='ubuntu@trial183.front.sepia.ceph.com')] 2026-02-25T08:08:49.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial169.front.sepia.ceph.com' 2026-02-25T08:08:49.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial169.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:08:49.151 DEBUG:teuthology.orchestra.run.trial169:> true 2026-02-25T08:08:49.227 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial169.front.sepia.ceph.com' 2026-02-25T08:08:49.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial183.front.sepia.ceph.com' 2026-02-25T08:08:49.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T08:08:49.323 DEBUG:teuthology.orchestra.run.trial183:> true 2026-02-25T08:08:49.401 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial183.front.sepia.ceph.com' 2026-02-25T08:08:49.401 INFO:teuthology.run_tasks:Running task clock... 2026-02-25T08:08:49.415 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-25T08:08:49.415 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T08:08:49.416 DEBUG:teuthology.orchestra.run.trial169:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:08:49.418 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T08:08:49.418 DEBUG:teuthology.orchestra.run.trial183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:08:49.444 INFO:teuthology.orchestra.run.trial169.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-25T08:08:49.457 INFO:teuthology.orchestra.run.trial169.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-25T08:08:49.469 INFO:teuthology.orchestra.run.trial183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-25T08:08:49.477 INFO:teuthology.orchestra.run.trial169.stderr:sudo: ntpd: command not found 2026-02-25T08:08:49.482 INFO:teuthology.orchestra.run.trial183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-25T08:08:49.487 INFO:teuthology.orchestra.run.trial169.stdout:506 Cannot talk to daemon 2026-02-25T08:08:49.499 INFO:teuthology.orchestra.run.trial169.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-25T08:08:49.503 INFO:teuthology.orchestra.run.trial183.stderr:sudo: ntpd: command not found 2026-02-25T08:08:49.510 INFO:teuthology.orchestra.run.trial169.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-25T08:08:49.512 INFO:teuthology.orchestra.run.trial183.stdout:506 Cannot talk to daemon 2026-02-25T08:08:49.524 INFO:teuthology.orchestra.run.trial183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-25T08:08:49.536 INFO:teuthology.orchestra.run.trial183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-25T08:08:49.556 INFO:teuthology.orchestra.run.trial169.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:08:49.582 INFO:teuthology.orchestra.run.trial183.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:08:49.584 INFO:teuthology.orchestra.run.trial169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:08:49.584 INFO:teuthology.orchestra.run.trial169.stdout:=============================================================================== 2026-02-25T08:08:49.585 INFO:teuthology.orchestra.run.trial169.stdout:^? time.richie.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:49.585 INFO:teuthology.orchestra.run.trial169.stdout:^? 44.190.5.123 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:49.585 INFO:teuthology.orchestra.run.trial169.stdout:^? time.cloudflare.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:49.585 INFO:teuthology.orchestra.run.trial169.stdout:^? us.timeserver.pub 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:49.609 INFO:teuthology.orchestra.run.trial183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:08:49.609 INFO:teuthology.orchestra.run.trial183.stdout:=============================================================================== 2026-02-25T08:08:49.609 INFO:teuthology.orchestra.run.trial183.stdout:^? 107-172-222-7-host.coloc> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:49.609 INFO:teuthology.orchestra.run.trial183.stdout:^? ntp.maxhost.io 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:49.609 INFO:teuthology.orchestra.run.trial183.stdout:^? rabendary.tesp.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:49.609 INFO:teuthology.orchestra.run.trial183.stdout:^? time.trtnw.net 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-25T08:08:49.611 INFO:teuthology.run_tasks:Running task install... 2026-02-25T08:08:49.624 DEBUG:teuthology.task.install:project ceph 2026-02-25T08:08:49.625 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}} 2026-02-25T08:08:49.625 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'} 2026-02-25T08:08:49.625 INFO:teuthology.task.install:Using flavor: debug 2026-02-25T08:08:49.634 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T08:08:49.635 INFO:teuthology.task.install:extra packages: {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']} 2026-02-25T08:08:49.635 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:08:49.636 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:08:49.640 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:08:49.641 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:08:49.845 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:08:49.845 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:08:49.855 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:08:49.856 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:08:50.200 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:08:50.201 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:08:50.201 DEBUG:teuthology.orchestra.run.trial169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:08:50.228 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-25T08:08:50.228 DEBUG:teuthology.orchestra.run.trial169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:08:50.252 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:08:50.252 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:08:50.252 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:08:50.278 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-osd-crimson, ceph-osd-crimson-debuginfo on remote rpm x86_64 2026-02-25T08:08:50.278 DEBUG:teuthology.orchestra.run.trial183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:08:50.292 DEBUG:teuthology.orchestra.run.trial169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:08:50.342 DEBUG:teuthology.orchestra.run.trial183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:08:50.366 DEBUG:teuthology.orchestra.run.trial169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:08:50.416 DEBUG:teuthology.orchestra.run.trial183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:08:50.429 INFO:teuthology.orchestra.run.trial169.stdout:check_obsoletes = 1 2026-02-25T08:08:50.430 DEBUG:teuthology.orchestra.run.trial169:> sudo yum clean all 2026-02-25T08:08:50.478 INFO:teuthology.orchestra.run.trial183.stdout:check_obsoletes = 1 2026-02-25T08:08:50.481 DEBUG:teuthology.orchestra.run.trial183:> sudo yum clean all 2026-02-25T08:08:50.578 INFO:teuthology.orchestra.run.trial169.stdout:41 files removed 2026-02-25T08:08:50.592 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-radosgw 2026-02-25T08:08:50.629 INFO:teuthology.orchestra.run.trial183.stdout:41 files removed 2026-02-25T08:08:50.644 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-radosgw 2026-02-25T08:08:51.507 INFO:teuthology.orchestra.run.trial183.stdout:ceph packages for x86_64 132 kB/s | 94 kB 00:00 2026-02-25T08:08:51.510 INFO:teuthology.orchestra.run.trial169.stdout:ceph packages for x86_64 122 kB/s | 94 kB 00:00 2026-02-25T08:08:52.174 INFO:teuthology.orchestra.run.trial183.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-25T08:08:52.183 INFO:teuthology.orchestra.run.trial169.stdout:ceph noarch packages 38 kB/s | 24 kB 00:00 2026-02-25T08:08:52.782 INFO:teuthology.orchestra.run.trial169.stdout:ceph source packages 3.6 kB/s | 2.1 kB 00:00 2026-02-25T08:08:52.793 INFO:teuthology.orchestra.run.trial183.stdout:ceph source packages 3.5 kB/s | 2.1 kB 00:00 2026-02-25T08:08:53.534 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.9 MB 00:00 2026-02-25T08:08:53.811 INFO:teuthology.orchestra.run.trial169.stdout:CentOS Stream 9 - BaseOS 8.8 MB/s | 8.9 MB 00:01 2026-02-25T08:08:57.293 INFO:teuthology.orchestra.run.trial169.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 27 MB 00:02 2026-02-25T08:09:01.361 INFO:teuthology.orchestra.run.trial169.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 8.0 MB 00:01 2026-02-25T08:09:04.643 INFO:teuthology.orchestra.run.trial169.stdout:CentOS Stream 9 - Extras packages 7.9 kB/s | 20 kB 00:02 2026-02-25T08:09:05.244 INFO:teuthology.orchestra.run.trial169.stdout:Extra Packages for Enterprise Linux 38 MB/s | 20 MB 00:00 2026-02-25T08:09:08.488 INFO:teuthology.orchestra.run.trial169.stdout:lab-extras 2.6 MB/s | 50 kB 00:00 2026-02-25T08:09:09.712 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:09:09.713 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:09:09.714 INFO:teuthology.orchestra.run.trial169.stdout: Package Arch Version Repository Size 2026-02-25T08:09:09.714 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:09:09.714 INFO:teuthology.orchestra.run.trial169.stdout:Installing: 2026-02-25T08:09:09.714 INFO:teuthology.orchestra.run.trial169.stdout: ceph-radosgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 11 M 2026-02-25T08:09:09.714 INFO:teuthology.orchestra.run.trial169.stdout:Upgrading: 2026-02-25T08:09:09.715 INFO:teuthology.orchestra.run.trial169.stdout: librados2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 3.3 M 2026-02-25T08:09:09.715 INFO:teuthology.orchestra.run.trial169.stdout: librbd1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.7 M 2026-02-25T08:09:09.715 INFO:teuthology.orchestra.run.trial169.stdout:Installing dependencies: 2026-02-25T08:09:09.715 INFO:teuthology.orchestra.run.trial169.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-25T08:09:09.715 INFO:teuthology.orchestra.run.trial169.stdout: ceph-base x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.1 M 2026-02-25T08:09:09.715 INFO:teuthology.orchestra.run.trial169.stdout: ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 21 M 2026-02-25T08:09:09.715 INFO:teuthology.orchestra.run.trial169.stdout: ceph-selinux x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 25 k 2026-02-25T08:09:09.716 INFO:teuthology.orchestra.run.trial169.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-25T08:09:09.716 INFO:teuthology.orchestra.run.trial169.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-25T08:09:09.716 INFO:teuthology.orchestra.run.trial169.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-25T08:09:09.716 INFO:teuthology.orchestra.run.trial169.stdout: libcephfs2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 860 k 2026-02-25T08:09:09.716 INFO:teuthology.orchestra.run.trial169.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-25T08:09:09.716 INFO:teuthology.orchestra.run.trial169.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-25T08:09:09.716 INFO:teuthology.orchestra.run.trial169.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-25T08:09:09.716 INFO:teuthology.orchestra.run.trial169.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-25T08:09:09.717 INFO:teuthology.orchestra.run.trial169.stdout: libradosstriper1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 263 k 2026-02-25T08:09:09.717 INFO:teuthology.orchestra.run.trial169.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-25T08:09:09.717 INFO:teuthology.orchestra.run.trial169.stdout: librgw2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.6 M 2026-02-25T08:09:09.717 INFO:teuthology.orchestra.run.trial169.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-25T08:09:09.717 INFO:teuthology.orchestra.run.trial169.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-25T08:09:09.717 INFO:teuthology.orchestra.run.trial169.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-25T08:09:09.717 INFO:teuthology.orchestra.run.trial169.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-25T08:09:09.717 INFO:teuthology.orchestra.run.trial169.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-25T08:09:09.717 INFO:teuthology.orchestra.run.trial169.stdout: python3-ceph-argparse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 45 k 2026-02-25T08:09:09.718 INFO:teuthology.orchestra.run.trial169.stdout: python3-ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 185 k 2026-02-25T08:09:09.718 INFO:teuthology.orchestra.run.trial169.stdout: python3-cephfs x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 204 k 2026-02-25T08:09:09.718 INFO:teuthology.orchestra.run.trial169.stdout: python3-rados x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 324 k 2026-02-25T08:09:09.718 INFO:teuthology.orchestra.run.trial169.stdout: python3-rbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 306 k 2026-02-25T08:09:09.718 INFO:teuthology.orchestra.run.trial169.stdout: python3-rgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 100 k 2026-02-25T08:09:09.718 INFO:teuthology.orchestra.run.trial169.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-25T08:09:09.718 INFO:teuthology.orchestra.run.trial169.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-25T08:09:09.718 INFO:teuthology.orchestra.run.trial169.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-25T08:09:09.718 INFO:teuthology.orchestra.run.trial169.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-25T08:09:09.719 INFO:teuthology.orchestra.run.trial169.stdout:Installing weak dependencies: 2026-02-25T08:09:09.719 INFO:teuthology.orchestra.run.trial169.stdout: libcephfs-daemon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:09:09.719 INFO:teuthology.orchestra.run.trial169.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 26 k 2026-02-25T08:09:09.719 INFO:teuthology.orchestra.run.trial169.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-25T08:09:09.719 INFO:teuthology.orchestra.run.trial169.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-25T08:09:09.719 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:09:09.719 INFO:teuthology.orchestra.run.trial169.stdout:Transaction Summary 2026-02-25T08:09:09.719 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:09:09.720 INFO:teuthology.orchestra.run.trial169.stdout:Install 35 Packages 2026-02-25T08:09:09.720 INFO:teuthology.orchestra.run.trial169.stdout:Upgrade 2 Packages 2026-02-25T08:09:09.720 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:09:09.721 INFO:teuthology.orchestra.run.trial169.stdout:Total download size: 61 M 2026-02-25T08:09:09.721 INFO:teuthology.orchestra.run.trial169.stdout:Downloading Packages: 2026-02-25T08:09:12.866 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 27 MB 00:18 2026-02-25T08:09:18.670 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 8.0 MB 00:03 2026-02-25T08:09:19.784 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 20 kB 00:00 2026-02-25T08:09:20.434 INFO:teuthology.orchestra.run.trial183.stdout:Extra Packages for Enterprise Linux 35 MB/s | 20 MB 00:00 2026-02-25T08:09:23.643 INFO:teuthology.orchestra.run.trial183.stdout:lab-extras 2.8 MB/s | 50 kB 00:00 2026-02-25T08:09:24.861 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:09:24.862 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:09:24.863 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repository Size 2026-02-25T08:09:24.863 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:09:24.863 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-25T08:09:24.863 INFO:teuthology.orchestra.run.trial183.stdout: ceph-radosgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 11 M 2026-02-25T08:09:24.863 INFO:teuthology.orchestra.run.trial183.stdout:Upgrading: 2026-02-25T08:09:24.863 INFO:teuthology.orchestra.run.trial183.stdout: librados2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 3.3 M 2026-02-25T08:09:24.863 INFO:teuthology.orchestra.run.trial183.stdout: librbd1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.7 M 2026-02-25T08:09:24.863 INFO:teuthology.orchestra.run.trial183.stdout:Installing dependencies: 2026-02-25T08:09:24.863 INFO:teuthology.orchestra.run.trial183.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-25T08:09:24.863 INFO:teuthology.orchestra.run.trial183.stdout: ceph-base x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.1 M 2026-02-25T08:09:24.863 INFO:teuthology.orchestra.run.trial183.stdout: ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 21 M 2026-02-25T08:09:24.864 INFO:teuthology.orchestra.run.trial183.stdout: ceph-selinux x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 25 k 2026-02-25T08:09:24.864 INFO:teuthology.orchestra.run.trial183.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-25T08:09:24.864 INFO:teuthology.orchestra.run.trial183.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-25T08:09:24.864 INFO:teuthology.orchestra.run.trial183.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-25T08:09:24.864 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 860 k 2026-02-25T08:09:24.864 INFO:teuthology.orchestra.run.trial183.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-25T08:09:24.864 INFO:teuthology.orchestra.run.trial183.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-25T08:09:24.864 INFO:teuthology.orchestra.run.trial183.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-25T08:09:24.864 INFO:teuthology.orchestra.run.trial183.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-25T08:09:24.864 INFO:teuthology.orchestra.run.trial183.stdout: libradosstriper1 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 263 k 2026-02-25T08:09:24.864 INFO:teuthology.orchestra.run.trial183.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-25T08:09:24.864 INFO:teuthology.orchestra.run.trial183.stdout: librgw2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 5.6 M 2026-02-25T08:09:24.865 INFO:teuthology.orchestra.run.trial183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-25T08:09:24.865 INFO:teuthology.orchestra.run.trial183.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-25T08:09:24.865 INFO:teuthology.orchestra.run.trial183.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-25T08:09:24.865 INFO:teuthology.orchestra.run.trial183.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-25T08:09:24.865 INFO:teuthology.orchestra.run.trial183.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-25T08:09:24.865 INFO:teuthology.orchestra.run.trial183.stdout: python3-ceph-argparse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 45 k 2026-02-25T08:09:24.865 INFO:teuthology.orchestra.run.trial183.stdout: python3-ceph-common x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 185 k 2026-02-25T08:09:24.865 INFO:teuthology.orchestra.run.trial183.stdout: python3-cephfs x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 204 k 2026-02-25T08:09:24.866 INFO:teuthology.orchestra.run.trial183.stdout: python3-rados x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 324 k 2026-02-25T08:09:24.866 INFO:teuthology.orchestra.run.trial183.stdout: python3-rbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 306 k 2026-02-25T08:09:24.866 INFO:teuthology.orchestra.run.trial183.stdout: python3-rgw x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 100 k 2026-02-25T08:09:24.866 INFO:teuthology.orchestra.run.trial183.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-25T08:09:24.866 INFO:teuthology.orchestra.run.trial183.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-25T08:09:24.866 INFO:teuthology.orchestra.run.trial183.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-25T08:09:24.866 INFO:teuthology.orchestra.run.trial183.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-25T08:09:24.866 INFO:teuthology.orchestra.run.trial183.stdout:Installing weak dependencies: 2026-02-25T08:09:24.866 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-daemon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:09:24.867 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 26 k 2026-02-25T08:09:24.867 INFO:teuthology.orchestra.run.trial183.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-25T08:09:24.867 INFO:teuthology.orchestra.run.trial183.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-25T08:09:24.867 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:09:24.867 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-25T08:09:24.867 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:09:24.868 INFO:teuthology.orchestra.run.trial183.stdout:Install 35 Packages 2026-02-25T08:09:24.868 INFO:teuthology.orchestra.run.trial183.stdout:Upgrade 2 Packages 2026-02-25T08:09:24.868 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:09:24.869 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 61 M 2026-02-25T08:09:24.869 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-25T08:09:30.145 INFO:teuthology.orchestra.run.trial169.stdout:(1/37): ceph-base-20.3.0-5338.g6aeb6975.el9.x86 261 kB/s | 5.1 MB 00:20 2026-02-25T08:09:30.287 INFO:teuthology.orchestra.run.trial169.stdout:(2/37): ceph-selinux-20.3.0-5338.g6aeb6975.el9. 177 kB/s | 25 kB 00:00 2026-02-25T08:09:30.496 INFO:teuthology.orchestra.run.trial169.stdout:(3/37): libcephfs-daemon-20.3.0-5338.g6aeb6975. 177 kB/s | 37 kB 00:00 2026-02-25T08:09:30.636 INFO:teuthology.orchestra.run.trial169.stdout:(4/37): libcephfs-proxy2-20.3.0-5338.g6aeb6975. 183 kB/s | 26 kB 00:00 2026-02-25T08:09:33.506 INFO:teuthology.orchestra.run.trial169.stdout:(5/37): libcephfs2-20.3.0-5338.g6aeb6975.el9.x8 300 kB/s | 860 kB 00:02 2026-02-25T08:09:34.147 INFO:teuthology.orchestra.run.trial169.stdout:(6/37): libradosstriper1-20.3.0-5338.g6aeb6975. 410 kB/s | 263 kB 00:00 2026-02-25T08:09:40.846 INFO:teuthology.orchestra.run.trial183.stdout:(1/37): ceph-base-20.3.0-5338.g6aeb6975.el9.x86 338 kB/s | 5.1 MB 00:15 2026-02-25T08:09:40.981 INFO:teuthology.orchestra.run.trial183.stdout:(2/37): ceph-selinux-20.3.0-5338.g6aeb6975.el9. 186 kB/s | 25 kB 00:00 2026-02-25T08:09:41.123 INFO:teuthology.orchestra.run.trial183.stdout:(3/37): libcephfs-daemon-20.3.0-5338.g6aeb6975. 262 kB/s | 37 kB 00:00 2026-02-25T08:09:41.262 INFO:teuthology.orchestra.run.trial183.stdout:(4/37): libcephfs-proxy2-20.3.0-5338.g6aeb6975. 183 kB/s | 26 kB 00:00 2026-02-25T08:09:44.482 INFO:teuthology.orchestra.run.trial183.stdout:(5/37): libcephfs2-20.3.0-5338.g6aeb6975.el9.x8 267 kB/s | 860 kB 00:03 2026-02-25T08:09:45.473 INFO:teuthology.orchestra.run.trial183.stdout:(6/37): libradosstriper1-20.3.0-5338.g6aeb6975. 265 kB/s | 263 kB 00:00 2026-02-25T08:09:46.436 INFO:teuthology.orchestra.run.trial169.stdout:(7/37): ceph-radosgw-20.3.0-5338.g6aeb6975.el9. 315 kB/s | 11 MB 00:36 2026-02-25T08:09:46.716 INFO:teuthology.orchestra.run.trial169.stdout:(8/37): python3-ceph-argparse-20.3.0-5338.g6aeb 161 kB/s | 45 kB 00:00 2026-02-25T08:09:47.742 INFO:teuthology.orchestra.run.trial169.stdout:(9/37): python3-ceph-common-20.3.0-5338.g6aeb69 180 kB/s | 185 kB 00:01 2026-02-25T08:09:48.558 INFO:teuthology.orchestra.run.trial169.stdout:(10/37): python3-cephfs-20.3.0-5338.g6aeb6975.e 250 kB/s | 204 kB 00:00 2026-02-25T08:09:50.198 INFO:teuthology.orchestra.run.trial169.stdout:(11/37): librgw2-20.3.0-5338.g6aeb6975.el9.x86_ 359 kB/s | 5.6 MB 00:16 2026-02-25T08:09:50.199 INFO:teuthology.orchestra.run.trial169.stdout:(12/37): python3-rados-20.3.0-5338.g6aeb6975.el 197 kB/s | 324 kB 00:01 2026-02-25T08:09:51.041 INFO:teuthology.orchestra.run.trial169.stdout:(13/37): python3-rgw-20.3.0-5338.g6aeb6975.el9. 119 kB/s | 100 kB 00:00 2026-02-25T08:09:51.121 INFO:teuthology.orchestra.run.trial169.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 412 kB/s | 33 kB 00:00 2026-02-25T08:09:51.221 INFO:teuthology.orchestra.run.trial169.stdout:(15/37): nvme-cli-2.16-1.el9.x86_64.rpm 12 MB/s | 1.2 MB 00:00 2026-02-25T08:09:51.254 INFO:teuthology.orchestra.run.trial169.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 2.8 MB/s | 93 kB 00:00 2026-02-25T08:09:51.378 INFO:teuthology.orchestra.run.trial169.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 837 kB/s | 104 kB 00:00 2026-02-25T08:09:51.434 INFO:teuthology.orchestra.run.trial169.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 2.9 MB/s | 164 kB 00:00 2026-02-25T08:09:51.469 INFO:teuthology.orchestra.run.trial169.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.5 MB/s | 160 kB 00:00 2026-02-25T08:09:51.496 INFO:teuthology.orchestra.run.trial169.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.6 MB/s | 45 kB 00:00 2026-02-25T08:09:51.565 INFO:teuthology.orchestra.run.trial169.stdout:(21/37): librdkafka-1.6.1-102.el9.x86_64.rpm 9.4 MB/s | 662 kB 00:00 2026-02-25T08:09:51.597 INFO:teuthology.orchestra.run.trial169.stdout:(22/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 9.2 MB/s | 292 kB 00:00 2026-02-25T08:09:51.626 INFO:teuthology.orchestra.run.trial169.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 8.0 MB/s | 240 kB 00:00 2026-02-25T08:09:51.651 INFO:teuthology.orchestra.run.trial169.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 1.4 MB/s | 37 kB 00:00 2026-02-25T08:09:51.677 INFO:teuthology.orchestra.run.trial169.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 2.5 MB/s | 66 kB 00:00 2026-02-25T08:09:51.989 INFO:teuthology.orchestra.run.trial169.stdout:(26/37): python3-rbd-20.3.0-5338.g6aeb6975.el9. 171 kB/s | 306 kB 00:01 2026-02-25T08:09:52.099 INFO:teuthology.orchestra.run.trial169.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 729 kB/s | 308 kB 00:00 2026-02-25T08:09:52.315 INFO:teuthology.orchestra.run.trial169.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 114 kB/s | 25 kB 00:00 2026-02-25T08:09:52.604 INFO:teuthology.orchestra.run.trial169.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 170 kB/s | 49 kB 00:00 2026-02-25T08:09:52.761 INFO:teuthology.orchestra.run.trial169.stdout:(30/37): libarrow-9.0.0-15.el9.x86_64.rpm 5.7 MB/s | 4.4 MB 00:00 2026-02-25T08:09:52.890 INFO:teuthology.orchestra.run.trial169.stdout:(31/37): libunwind-1.6.2-1.el9.x86_64.rpm 235 kB/s | 67 kB 00:00 2026-02-25T08:09:53.249 INFO:teuthology.orchestra.run.trial169.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 534 kB/s | 191 kB 00:00 2026-02-25T08:09:53.252 INFO:teuthology.orchestra.run.trial169.stdout:(33/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 1.7 MB/s | 838 kB 00:00 2026-02-25T08:09:53.821 INFO:teuthology.orchestra.run.trial169.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 2.8 MB/s | 1.6 MB 00:00 2026-02-25T08:10:05.994 INFO:teuthology.orchestra.run.trial169.stdout:(35/37): librbd1-20.3.0-5338.g6aeb6975.el9.x86_ 226 kB/s | 2.7 MB 00:12 2026-02-25T08:10:08.010 INFO:teuthology.orchestra.run.trial169.stdout:(36/37): librados2-20.3.0-5338.g6aeb6975.el9.x8 228 kB/s | 3.3 MB 00:14 2026-02-25T08:10:10.852 INFO:teuthology.orchestra.run.trial169.stdout:(37/37): ceph-common-20.3.0-5338.g6aeb6975.el9. 351 kB/s | 21 MB 01:00 2026-02-25T08:10:10.854 INFO:teuthology.orchestra.run.trial169.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:10:10.854 INFO:teuthology.orchestra.run.trial169.stdout:Total 1.0 MB/s | 61 MB 01:01 2026-02-25T08:10:10.944 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction check 2026-02-25T08:10:10.963 INFO:teuthology.orchestra.run.trial169.stdout:Transaction check succeeded. 2026-02-25T08:10:10.963 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction test 2026-02-25T08:10:10.986 INFO:teuthology.orchestra.run.trial183.stdout:(7/37): ceph-radosgw-20.3.0-5338.g6aeb6975.el9. 251 kB/s | 11 MB 00:45 2026-02-25T08:10:11.196 INFO:teuthology.orchestra.run.trial183.stdout:(8/37): python3-ceph-argparse-20.3.0-5338.g6aeb 215 kB/s | 45 kB 00:00 2026-02-25T08:10:11.211 INFO:teuthology.orchestra.run.trial169.stdout:Transaction test succeeded. 2026-02-25T08:10:11.211 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction 2026-02-25T08:10:11.503 INFO:teuthology.orchestra.run.trial169.stdout: Preparing : 1/1 2026-02-25T08:10:11.644 INFO:teuthology.orchestra.run.trial169.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-25T08:10:11.649 INFO:teuthology.orchestra.run.trial169.stdout: Upgrading : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:10:11.697 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:10:11.698 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:10:11.710 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:10:11.712 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 4/39 2026-02-25T08:10:11.713 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:10:11.740 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:10:11.743 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 6/39 2026-02-25T08:10:11.750 INFO:teuthology.orchestra.run.trial169.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-25T08:10:11.753 INFO:teuthology.orchestra.run.trial169.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-25T08:10:11.756 INFO:teuthology.orchestra.run.trial169.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-25T08:10:11.760 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-25T08:10:11.771 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 11/39 2026-02-25T08:10:11.784 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:10:11.784 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:10:11.807 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:10:11.812 INFO:teuthology.orchestra.run.trial169.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-25T08:10:11.844 INFO:teuthology.orchestra.run.trial169.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-25T08:10:11.848 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-25T08:10:11.855 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-25T08:10:11.857 INFO:teuthology.orchestra.run.trial169.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-25T08:10:11.861 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-25T08:10:11.975 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:10:11.976 INFO:teuthology.orchestra.run.trial169.stdout: Upgrading : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:10:12.002 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:10:12.005 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 22/39 2026-02-25T08:10:12.013 INFO:teuthology.orchestra.run.trial169.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-25T08:10:12.096 INFO:teuthology.orchestra.run.trial183.stdout:(9/37): python3-ceph-common-20.3.0-5338.g6aeb69 205 kB/s | 185 kB 00:00 2026-02-25T08:10:12.261 INFO:teuthology.orchestra.run.trial169.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-25T08:10:12.262 INFO:teuthology.orchestra.run.trial169.stdout: Installing : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:10:12.278 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:10:12.282 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 26/39 2026-02-25T08:10:12.287 INFO:teuthology.orchestra.run.trial169.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-25T08:10:12.290 INFO:teuthology.orchestra.run.trial169.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-25T08:10:12.290 INFO:teuthology.orchestra.run.trial169.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:10:12.317 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:10:12.430 INFO:teuthology.orchestra.run.trial169.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-25T08:10:12.433 INFO:teuthology.orchestra.run.trial169.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:10:12.713 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:10:12.713 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T08:10:12.713 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:10:12.756 INFO:teuthology.orchestra.run.trial169.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-25T08:10:12.758 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 33/39 2026-02-25T08:10:12.871 INFO:teuthology.orchestra.run.trial183.stdout:(10/37): python3-cephfs-20.3.0-5338.g6aeb6975.e 263 kB/s | 204 kB 00:00 2026-02-25T08:10:13.701 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:10:13.704 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:10:13.963 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:10:13.966 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:10:14.002 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:10:14.002 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-25T08:10:14.003 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-25T08:10:14.003 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:10:14.006 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:10:14.340 INFO:teuthology.orchestra.run.trial183.stdout:(11/37): python3-rados-20.3.0-5338.g6aeb6975.el 221 kB/s | 324 kB 00:01 2026-02-25T08:10:14.517 INFO:teuthology.orchestra.run.trial183.stdout:(12/37): librgw2-20.3.0-5338.g6aeb6975.el9.x86_ 198 kB/s | 5.6 MB 00:29 2026-02-25T08:10:15.217 INFO:teuthology.orchestra.run.trial183.stdout:(13/37): python3-rgw-20.3.0-5338.g6aeb6975.el9. 143 kB/s | 100 kB 00:00 2026-02-25T08:10:15.294 INFO:teuthology.orchestra.run.trial183.stdout:(14/37): mailcap-2.1.49-5.el9.noarch.rpm 433 kB/s | 33 kB 00:00 2026-02-25T08:10:15.388 INFO:teuthology.orchestra.run.trial183.stdout:(15/37): nvme-cli-2.16-1.el9.x86_64.rpm 12 MB/s | 1.2 MB 00:00 2026-02-25T08:10:15.408 INFO:teuthology.orchestra.run.trial183.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 4.5 MB/s | 93 kB 00:00 2026-02-25T08:10:15.429 INFO:teuthology.orchestra.run.trial183.stdout:(17/37): boost-program-options-1.75.0-13.el9.x8 5.0 MB/s | 104 kB 00:00 2026-02-25T08:10:15.450 INFO:teuthology.orchestra.run.trial183.stdout:(18/37): libnbd-1.20.3-4.el9.x86_64.rpm 7.6 MB/s | 164 kB 00:00 2026-02-25T08:10:15.471 INFO:teuthology.orchestra.run.trial183.stdout:(19/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 7.4 MB/s | 160 kB 00:00 2026-02-25T08:10:15.491 INFO:teuthology.orchestra.run.trial183.stdout:(20/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 2.2 MB/s | 45 kB 00:00 2026-02-25T08:10:15.513 INFO:teuthology.orchestra.run.trial183.stdout:(21/37): librdkafka-1.6.1-102.el9.x86_64.rpm 29 MB/s | 662 kB 00:00 2026-02-25T08:10:15.530 INFO:teuthology.orchestra.run.trial183.stdout:(22/37): python3-rbd-20.3.0-5338.g6aeb6975.el9. 257 kB/s | 306 kB 00:01 2026-02-25T08:10:15.534 INFO:teuthology.orchestra.run.trial183.stdout:(23/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 13 MB/s | 292 kB 00:00 2026-02-25T08:10:15.554 INFO:teuthology.orchestra.run.trial183.stdout:(24/37): qatlib-service-25.08.0-2.el9.x86_64.rp 1.9 MB/s | 37 kB 00:00 2026-02-25T08:10:15.574 INFO:teuthology.orchestra.run.trial183.stdout:(25/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 3.3 MB/s | 66 kB 00:00 2026-02-25T08:10:15.646 INFO:teuthology.orchestra.run.trial183.stdout:(26/37): qatlib-25.08.0-2.el9.x86_64.rpm 2.0 MB/s | 240 kB 00:00 2026-02-25T08:10:18.509 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:10:18.509 INFO:teuthology.orchestra.run.trial169.stdout:skipping the directory /sys 2026-02-25T08:10:18.510 INFO:teuthology.orchestra.run.trial169.stdout:skipping the directory /proc 2026-02-25T08:10:18.510 INFO:teuthology.orchestra.run.trial169.stdout:skipping the directory /mnt 2026-02-25T08:10:18.510 INFO:teuthology.orchestra.run.trial169.stdout:skipping the directory /var/tmp 2026-02-25T08:10:18.510 INFO:teuthology.orchestra.run.trial169.stdout:skipping the directory /home 2026-02-25T08:10:18.510 INFO:teuthology.orchestra.run.trial169.stdout:skipping the directory /root 2026-02-25T08:10:18.510 INFO:teuthology.orchestra.run.trial169.stdout:skipping the directory /tmp 2026-02-25T08:10:18.510 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:10:19.003 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:10:19.026 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:10:19.026 INFO:teuthology.orchestra.run.trial169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:10:19.026 INFO:teuthology.orchestra.run.trial169.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-25T08:10:19.026 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:10:19.026 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:10:19.026 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:10:19.028 INFO:teuthology.orchestra.run.trial169.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:10:19.040 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:10:19.040 INFO:teuthology.orchestra.run.trial169.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:10:19.093 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:10:19.094 INFO:teuthology.orchestra.run.trial169.stdout:Creating group 'qat' with GID 992. 2026-02-25T08:10:19.094 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:10:19.381 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/39 2026-02-25T08:10:19.381 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:10:19.381 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:10:19.381 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/39 2026-02-25T08:10:19.381 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:10:19.381 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/39 2026-02-25T08:10:19.381 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 7/39 2026-02-25T08:10:19.381 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 8/39 2026-02-25T08:10:19.381 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 9/39 2026-02-25T08:10:19.381 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 10/39 2026-02-25T08:10:19.382 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 11/39 2026-02-25T08:10:19.382 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:10:19.382 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 13/39 2026-02-25T08:10:19.382 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 14/39 2026-02-25T08:10:19.382 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 15/39 2026-02-25T08:10:19.382 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-25T08:10:19.382 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-25T08:10:19.382 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-25T08:10:19.382 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-25T08:10:19.382 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:10:19.382 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-25T08:10:19.382 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-25T08:10:19.382 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-25T08:10:19.383 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-25T08:10:19.383 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-25T08:10:19.383 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-25T08:10:19.383 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-25T08:10:19.383 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-25T08:10:19.383 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-25T08:10:19.383 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-25T08:10:19.383 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-25T08:10:19.383 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-25T08:10:19.383 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-25T08:10:19.383 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-25T08:10:19.383 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-25T08:10:19.384 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:10:19.384 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-25T08:10:19.384 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 38/39 2026-02-25T08:10:19.452 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:10:19.452 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:10:19.452 INFO:teuthology.orchestra.run.trial169.stdout:Upgraded: 2026-02-25T08:10:19.452 INFO:teuthology.orchestra.run.trial169.stdout: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.452 INFO:teuthology.orchestra.run.trial169.stdout: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.452 INFO:teuthology.orchestra.run.trial169.stdout:Installed: 2026-02-25T08:10:19.452 INFO:teuthology.orchestra.run.trial169.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-25T08:10:19.452 INFO:teuthology.orchestra.run.trial169.stdout: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.453 INFO:teuthology.orchestra.run.trial169.stdout: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.453 INFO:teuthology.orchestra.run.trial169.stdout: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.453 INFO:teuthology.orchestra.run.trial169.stdout: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.453 INFO:teuthology.orchestra.run.trial169.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-25T08:10:19.453 INFO:teuthology.orchestra.run.trial169.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-25T08:10:19.453 INFO:teuthology.orchestra.run.trial169.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-25T08:10:19.453 INFO:teuthology.orchestra.run.trial169.stdout: libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.453 INFO:teuthology.orchestra.run.trial169.stdout: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.453 INFO:teuthology.orchestra.run.trial169.stdout: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.453 INFO:teuthology.orchestra.run.trial169.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-25T08:10:19.453 INFO:teuthology.orchestra.run.trial169.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-25T08:10:19.453 INFO:teuthology.orchestra.run.trial169.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-25T08:10:19.454 INFO:teuthology.orchestra.run.trial169.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-25T08:10:19.454 INFO:teuthology.orchestra.run.trial169.stdout: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.454 INFO:teuthology.orchestra.run.trial169.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-25T08:10:19.454 INFO:teuthology.orchestra.run.trial169.stdout: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.454 INFO:teuthology.orchestra.run.trial169.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-25T08:10:19.454 INFO:teuthology.orchestra.run.trial169.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-25T08:10:19.454 INFO:teuthology.orchestra.run.trial169.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-25T08:10:19.454 INFO:teuthology.orchestra.run.trial169.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-25T08:10:19.454 INFO:teuthology.orchestra.run.trial169.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-25T08:10:19.454 INFO:teuthology.orchestra.run.trial169.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-25T08:10:19.454 INFO:teuthology.orchestra.run.trial169.stdout: python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.454 INFO:teuthology.orchestra.run.trial169.stdout: python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.454 INFO:teuthology.orchestra.run.trial169.stdout: python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.455 INFO:teuthology.orchestra.run.trial169.stdout: python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.455 INFO:teuthology.orchestra.run.trial169.stdout: python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.455 INFO:teuthology.orchestra.run.trial169.stdout: python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:10:19.455 INFO:teuthology.orchestra.run.trial169.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-25T08:10:19.455 INFO:teuthology.orchestra.run.trial169.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-25T08:10:19.455 INFO:teuthology.orchestra.run.trial169.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-25T08:10:19.455 INFO:teuthology.orchestra.run.trial169.stdout: re2-1:20211101-20.el9.x86_64 2026-02-25T08:10:19.455 INFO:teuthology.orchestra.run.trial169.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-25T08:10:19.455 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:10:19.455 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:10:19.492 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-test 2026-02-25T08:10:19.808 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:01:11 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:10:19.891 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:10:19.892 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:10:19.892 INFO:teuthology.orchestra.run.trial169.stdout: Package Arch Version Repository Size 2026-02-25T08:10:19.892 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:10:19.892 INFO:teuthology.orchestra.run.trial169.stdout:Installing: 2026-02-25T08:10:19.892 INFO:teuthology.orchestra.run.trial169.stdout: ceph-test x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 127 M 2026-02-25T08:10:19.892 INFO:teuthology.orchestra.run.trial169.stdout:Installing dependencies: 2026-02-25T08:10:19.892 INFO:teuthology.orchestra.run.trial169.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-25T08:10:19.892 INFO:teuthology.orchestra.run.trial169.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-25T08:10:19.892 INFO:teuthology.orchestra.run.trial169.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-25T08:10:19.892 INFO:teuthology.orchestra.run.trial169.stdout: libcephsqlite x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 153 k 2026-02-25T08:10:19.892 INFO:teuthology.orchestra.run.trial169.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-25T08:10:19.892 INFO:teuthology.orchestra.run.trial169.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-25T08:10:19.892 INFO:teuthology.orchestra.run.trial169.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-25T08:10:19.893 INFO:teuthology.orchestra.run.trial169.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-25T08:10:19.893 INFO:teuthology.orchestra.run.trial169.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-25T08:10:19.893 INFO:teuthology.orchestra.run.trial169.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-25T08:10:19.893 INFO:teuthology.orchestra.run.trial169.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-25T08:10:19.893 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:10:19.893 INFO:teuthology.orchestra.run.trial169.stdout:Transaction Summary 2026-02-25T08:10:19.893 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:10:19.893 INFO:teuthology.orchestra.run.trial169.stdout:Install 12 Packages 2026-02-25T08:10:19.893 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:10:19.893 INFO:teuthology.orchestra.run.trial169.stdout:Total download size: 132 M 2026-02-25T08:10:19.894 INFO:teuthology.orchestra.run.trial169.stdout:Installed size: 988 M 2026-02-25T08:10:19.894 INFO:teuthology.orchestra.run.trial169.stdout:Downloading Packages: 2026-02-25T08:10:21.392 INFO:teuthology.orchestra.run.trial169.stdout:(1/12): libcephsqlite-20.3.0-5338.g6aeb6975.el9 154 kB/s | 153 kB 00:00 2026-02-25T08:10:21.462 INFO:teuthology.orchestra.run.trial169.stdout:(2/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 2.0 MB/s | 2.1 MB 00:01 2026-02-25T08:10:21.555 INFO:teuthology.orchestra.run.trial169.stdout:(3/12): libatomic-11.5.0-14.el9.x86_64.rpm 139 kB/s | 23 kB 00:00 2026-02-25T08:10:21.679 INFO:teuthology.orchestra.run.trial169.stdout:(4/12): libasan8-15.2.1-7.el9.x86_64.rpm 4.2 MB/s | 525 kB 00:00 2026-02-25T08:10:21.781 INFO:teuthology.orchestra.run.trial169.stdout:(5/12): libubsan-11.5.0-14.el9.x86_64.rpm 1.7 MB/s | 176 kB 00:00 2026-02-25T08:10:21.884 INFO:teuthology.orchestra.run.trial169.stdout:(6/12): libxslt-1.1.34-12.el9.x86_64.rpm 2.2 MB/s | 233 kB 00:00 2026-02-25T08:10:21.990 INFO:teuthology.orchestra.run.trial169.stdout:(7/12): protobuf-3.14.0-17.el9.x86_64.rpm 9.4 MB/s | 1.0 MB 00:00 2026-02-25T08:10:22.094 INFO:teuthology.orchestra.run.trial169.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 2.9 MB/s | 303 kB 00:00 2026-02-25T08:10:22.195 INFO:teuthology.orchestra.run.trial169.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 627 kB/s | 64 kB 00:00 2026-02-25T08:10:22.201 INFO:teuthology.orchestra.run.trial169.stdout:(10/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 127 kB/s | 94 kB 00:00 2026-02-25T08:10:22.363 INFO:teuthology.orchestra.run.trial169.stdout:(11/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 725 kB/s | 122 kB 00:00 2026-02-25T08:10:35.794 INFO:teuthology.orchestra.run.trial183.stdout:(27/37): ceph-common-20.3.0-5338.g6aeb6975.el9. 303 kB/s | 21 MB 01:10 2026-02-25T08:12:27.298 INFO:teuthology.orchestra.run.trial183.stdout:[MIRROR] gperftools-libs-2.9.1-3.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://cfhcable.mm.fcix.net/epel/9/Everything/x86_64/Packages/g/gperftools-libs-2.9.1-3.el9.x86_64.rpm [Failed to connect to cfhcable.mm.fcix.net port 443: Connection timed out] 2026-02-25T08:12:27.304 INFO:teuthology.orchestra.run.trial183.stdout:[MIRROR] libarrow-9.0.0-15.el9.x86_64.rpm: Curl error (28): Timeout was reached for https://cfhcable.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-9.0.0-15.el9.x86_64.rpm [Failed to connect to cfhcable.mm.fcix.net port 443: Connection timed out] 2026-02-25T08:12:27.558 INFO:teuthology.orchestra.run.trial183.stdout:(28/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.3 kB/s | 308 kB 02:11 2026-02-25T08:12:27.596 INFO:teuthology.orchestra.run.trial183.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 1.3 MB/s | 49 kB 00:00 2026-02-25T08:12:27.635 INFO:teuthology.orchestra.run.trial183.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 1.7 MB/s | 67 kB 00:00 2026-02-25T08:12:27.696 INFO:teuthology.orchestra.run.trial183.stdout:(31/37): libarrow-9.0.0-15.el9.x86_64.rpm 34 kB/s | 4.4 MB 02:12 2026-02-25T08:12:27.734 INFO:teuthology.orchestra.run.trial183.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 5.0 MB/s | 191 kB 00:00 2026-02-25T08:12:27.735 INFO:teuthology.orchestra.run.trial183.stdout:(33/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2026-02-25T08:12:27.820 INFO:teuthology.orchestra.run.trial183.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 18 MB/s | 1.6 MB 00:00 2026-02-25T08:12:31.105 INFO:teuthology.orchestra.run.trial183.stdout:(35/37): librbd1-20.3.0-5338.g6aeb6975.el9.x86_ 838 kB/s | 2.7 MB 00:03 2026-02-25T08:12:33.435 INFO:teuthology.orchestra.run.trial183.stdout:(36/37): librados2-20.3.0-5338.g6aeb6975.el9.x8 590 kB/s | 3.3 MB 00:05 2026-02-25T08:12:47.943 INFO:teuthology.orchestra.run.trial183.stdout:[MIRROR] libarrow-doc-9.0.0-15.el9.noarch.rpm: Curl error (28): Timeout was reached for https://cfhcable.mm.fcix.net/epel/9/Everything/x86_64/Packages/l/libarrow-doc-9.0.0-15.el9.noarch.rpm [Failed to connect to cfhcable.mm.fcix.net port 443: Connection timed out] 2026-02-25T08:12:48.054 INFO:teuthology.orchestra.run.trial183.stdout:(37/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 192 B/s | 25 kB 02:12 2026-02-25T08:12:48.056 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:12:48.056 INFO:teuthology.orchestra.run.trial183.stdout:Total 310 kB/s | 61 MB 03:23 2026-02-25T08:12:48.149 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-25T08:12:48.168 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-25T08:12:48.168 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-25T08:12:48.416 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-25T08:12:48.416 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-25T08:12:48.708 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-25T08:12:48.849 INFO:teuthology.orchestra.run.trial183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-25T08:12:48.853 INFO:teuthology.orchestra.run.trial183.stdout: Upgrading : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:12:48.903 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:12:48.904 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:12:48.916 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:12:48.918 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 4/39 2026-02-25T08:12:48.918 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:12:48.945 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:12:48.947 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 6/39 2026-02-25T08:12:48.956 INFO:teuthology.orchestra.run.trial183.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-25T08:12:48.958 INFO:teuthology.orchestra.run.trial183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-25T08:12:48.962 INFO:teuthology.orchestra.run.trial183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-25T08:12:48.966 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-25T08:12:48.977 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 11/39 2026-02-25T08:12:48.990 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:12:48.990 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:12:49.011 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 13/39 2026-02-25T08:12:49.015 INFO:teuthology.orchestra.run.trial183.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-25T08:12:49.046 INFO:teuthology.orchestra.run.trial183.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-25T08:12:49.050 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-25T08:12:49.057 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-25T08:12:49.059 INFO:teuthology.orchestra.run.trial183.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-25T08:12:49.063 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-25T08:12:49.176 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:12:49.177 INFO:teuthology.orchestra.run.trial183.stdout: Upgrading : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:12:49.205 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 21/39 2026-02-25T08:12:49.207 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 22/39 2026-02-25T08:12:49.214 INFO:teuthology.orchestra.run.trial183.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-25T08:12:49.461 INFO:teuthology.orchestra.run.trial183.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-25T08:12:49.462 INFO:teuthology.orchestra.run.trial183.stdout: Installing : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:12:49.478 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 25/39 2026-02-25T08:12:49.483 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 26/39 2026-02-25T08:12:49.489 INFO:teuthology.orchestra.run.trial183.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-25T08:12:49.491 INFO:teuthology.orchestra.run.trial183.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-25T08:12:49.492 INFO:teuthology.orchestra.run.trial183.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:12:49.509 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-25T08:12:49.627 INFO:teuthology.orchestra.run.trial183.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-25T08:12:49.629 INFO:teuthology.orchestra.run.trial183.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:12:49.907 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-25T08:12:49.907 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T08:12:49.907 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:12:49.951 INFO:teuthology.orchestra.run.trial183.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-25T08:12:49.953 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 33/39 2026-02-25T08:12:50.893 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:12:50.896 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:12:51.161 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 34/39 2026-02-25T08:12:51.164 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:12:51.202 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 35/39 2026-02-25T08:12:51.202 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-25T08:12:51.203 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-25T08:12:51.203 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:12:51.206 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:12:55.703 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:12:55.703 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /sys 2026-02-25T08:12:55.704 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /proc 2026-02-25T08:12:55.704 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /mnt 2026-02-25T08:12:55.704 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /var/tmp 2026-02-25T08:12:55.704 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /home 2026-02-25T08:12:55.704 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /root 2026-02-25T08:12:55.704 INFO:teuthology.orchestra.run.trial183.stdout:skipping the directory /tmp 2026-02-25T08:12:55.704 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:12:56.195 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:12:56.218 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 37/39 2026-02-25T08:12:56.218 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:12:56.218 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-25T08:12:56.219 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:12:56.219 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-25T08:12:56.219 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:12:56.220 INFO:teuthology.orchestra.run.trial183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:12:56.231 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-25T08:12:56.232 INFO:teuthology.orchestra.run.trial183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:12:56.284 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:12:56.285 INFO:teuthology.orchestra.run.trial183.stdout:Creating group 'qat' with GID 992. 2026-02-25T08:12:56.285 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:12:56.573 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/39 2026-02-25T08:12:56.573 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/39 2026-02-25T08:12:56.574 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/39 2026-02-25T08:12:56.574 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/39 2026-02-25T08:12:56.574 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_6 5/39 2026-02-25T08:12:56.574 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/39 2026-02-25T08:12:56.574 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 7/39 2026-02-25T08:12:56.574 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_6 8/39 2026-02-25T08:12:56.574 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 9/39 2026-02-25T08:12:56.574 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9. 10/39 2026-02-25T08:12:56.574 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x8 11/39 2026-02-25T08:12:56.574 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/39 2026-02-25T08:12:56.575 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 13/39 2026-02-25T08:12:56.575 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 14/39 2026-02-25T08:12:56.575 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 15/39 2026-02-25T08:12:56.575 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-25T08:12:56.575 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-25T08:12:56.575 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-25T08:12:56.575 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-25T08:12:56.575 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-25T08:12:56.575 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-25T08:12:56.575 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-25T08:12:56.575 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-25T08:12:56.575 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-25T08:12:56.576 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-25T08:12:56.576 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-25T08:12:56.576 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-25T08:12:56.576 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-25T08:12:56.576 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-25T08:12:56.576 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-25T08:12:56.576 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-25T08:12:56.576 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-25T08:12:56.576 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-25T08:12:56.576 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-25T08:12:56.576 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-25T08:12:56.576 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 36/39 2026-02-25T08:12:56.577 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-25T08:12:56.577 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 38/39 2026-02-25T08:12:56.642 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-25T08:12:56.642 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:12:56.642 INFO:teuthology.orchestra.run.trial183.stdout:Upgraded: 2026-02-25T08:12:56.642 INFO:teuthology.orchestra.run.trial183.stdout: librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.642 INFO:teuthology.orchestra.run.trial183.stdout: librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.642 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-25T08:12:56.642 INFO:teuthology.orchestra.run.trial183.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-25T08:12:56.643 INFO:teuthology.orchestra.run.trial183.stdout: ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.643 INFO:teuthology.orchestra.run.trial183.stdout: ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.643 INFO:teuthology.orchestra.run.trial183.stdout: ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.643 INFO:teuthology.orchestra.run.trial183.stdout: ceph-selinux-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.643 INFO:teuthology.orchestra.run.trial183.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-25T08:12:56.643 INFO:teuthology.orchestra.run.trial183.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-25T08:12:56.643 INFO:teuthology.orchestra.run.trial183.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-25T08:12:56.643 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-daemon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.643 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-proxy2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.643 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.643 INFO:teuthology.orchestra.run.trial183.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-25T08:12:56.643 INFO:teuthology.orchestra.run.trial183.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-25T08:12:56.644 INFO:teuthology.orchestra.run.trial183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-25T08:12:56.644 INFO:teuthology.orchestra.run.trial183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-25T08:12:56.644 INFO:teuthology.orchestra.run.trial183.stdout: libradosstriper1-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.644 INFO:teuthology.orchestra.run.trial183.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-25T08:12:56.644 INFO:teuthology.orchestra.run.trial183.stdout: librgw2-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.644 INFO:teuthology.orchestra.run.trial183.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-25T08:12:56.644 INFO:teuthology.orchestra.run.trial183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-25T08:12:56.644 INFO:teuthology.orchestra.run.trial183.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-25T08:12:56.644 INFO:teuthology.orchestra.run.trial183.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-25T08:12:56.644 INFO:teuthology.orchestra.run.trial183.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-25T08:12:56.644 INFO:teuthology.orchestra.run.trial183.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-25T08:12:56.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-ceph-argparse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-ceph-common-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:12:56.645 INFO:teuthology.orchestra.run.trial183.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-25T08:12:56.645 INFO:teuthology.orchestra.run.trial183.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-25T08:12:56.645 INFO:teuthology.orchestra.run.trial183.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-25T08:12:56.645 INFO:teuthology.orchestra.run.trial183.stdout: re2-1:20211101-20.el9.x86_64 2026-02-25T08:12:56.645 INFO:teuthology.orchestra.run.trial183.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-25T08:12:56.645 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:12:56.646 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:12:56.683 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-test 2026-02-25T08:12:57.002 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:03:34 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:12:57.085 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repository Size 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: ceph-test x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 127 M 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout:Installing dependencies: 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: libasan8 x86_64 15.2.1-7.el9 appstream 525 k 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: libcephsqlite x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 153 k 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: libubsan x86_64 11.5.0-14.el9 appstream 176 k 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout:Install 12 Packages 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 132 M 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 988 M 2026-02-25T08:12:57.086 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-25T08:12:57.514 INFO:teuthology.orchestra.run.trial183.stdout:(1/12): hwloc-libs-2.4.1-6.el9.x86_64.rpm 10 MB/s | 2.1 MB 00:00 2026-02-25T08:12:57.534 INFO:teuthology.orchestra.run.trial183.stdout:(2/12): libatomic-11.5.0-14.el9.x86_64.rpm 1.1 MB/s | 23 kB 00:00 2026-02-25T08:12:57.555 INFO:teuthology.orchestra.run.trial183.stdout:(3/12): lksctp-tools-1.0.19-2.el9.x86_64.rpm 4.5 MB/s | 94 kB 00:00 2026-02-25T08:12:58.116 INFO:teuthology.orchestra.run.trial183.stdout:(4/12): libcephsqlite-20.3.0-5338.g6aeb6975.el9 190 kB/s | 153 kB 00:00 2026-02-25T08:12:58.642 INFO:teuthology.orchestra.run.trial183.stdout:(5/12): libasan8-15.2.1-7.el9.x86_64.rpm 483 kB/s | 525 kB 00:01 2026-02-25T08:12:58.784 INFO:teuthology.orchestra.run.trial183.stdout:(6/12): libubsan-11.5.0-14.el9.x86_64.rpm 264 kB/s | 176 kB 00:00 2026-02-25T08:12:58.810 INFO:teuthology.orchestra.run.trial183.stdout:(7/12): libxslt-1.1.34-12.el9.x86_64.rpm 1.4 MB/s | 233 kB 00:00 2026-02-25T08:12:58.898 INFO:teuthology.orchestra.run.trial183.stdout:(8/12): socat-1.7.4.1-8.el9.x86_64.rpm 3.4 MB/s | 303 kB 00:00 2026-02-25T08:12:58.984 INFO:teuthology.orchestra.run.trial183.stdout:(9/12): xmlstarlet-1.6.1-20.el9.x86_64.rpm 740 kB/s | 64 kB 00:00 2026-02-25T08:12:59.156 INFO:teuthology.orchestra.run.trial183.stdout:(10/12): yaml-cpp-0.6.3-5.el9.x86_64.rpm 708 kB/s | 122 kB 00:00 2026-02-25T08:13:00.017 INFO:teuthology.orchestra.run.trial183.stdout:(11/12): protobuf-3.14.0-17.el9.x86_64.rpm 834 kB/s | 1.0 MB 00:01 2026-02-25T08:18:01.492 INFO:teuthology.orchestra.run.trial169.stdout:(12/12): ceph-test-20.3.0-5338.g6aeb6975.el9.x8 282 kB/s | 127 MB 07:41 2026-02-25T08:18:01.499 INFO:teuthology.orchestra.run.trial169.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:18:01.499 INFO:teuthology.orchestra.run.trial169.stdout:Total 292 kB/s | 132 MB 07:41 2026-02-25T08:18:01.565 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction check 2026-02-25T08:18:01.578 INFO:teuthology.orchestra.run.trial169.stdout:Transaction check succeeded. 2026-02-25T08:18:01.578 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction test 2026-02-25T08:18:02.011 INFO:teuthology.orchestra.run.trial169.stdout:Transaction test succeeded. 2026-02-25T08:18:02.012 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction 2026-02-25T08:18:02.471 INFO:teuthology.orchestra.run.trial169.stdout: Preparing : 1/1 2026-02-25T08:18:02.480 INFO:teuthology.orchestra.run.trial169.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-25T08:18:02.488 INFO:teuthology.orchestra.run.trial169.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-25T08:18:02.493 INFO:teuthology.orchestra.run.trial169.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-25T08:18:02.497 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-25T08:18:02.500 INFO:teuthology.orchestra.run.trial169.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-25T08:18:02.504 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-25T08:18:02.505 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:18:02.522 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:18:02.524 INFO:teuthology.orchestra.run.trial169.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-25T08:18:02.535 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-25T08:18:02.544 INFO:teuthology.orchestra.run.trial169.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-25T08:18:02.545 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:18:09.494 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:18:09.513 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:18:10.287 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:18:10.288 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/12 2026-02-25T08:18:10.288 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/12 2026-02-25T08:18:10.288 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-25T08:18:10.288 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-25T08:18:10.288 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-25T08:18:10.288 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-25T08:18:10.288 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-25T08:18:10.288 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-25T08:18:10.288 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-25T08:18:10.288 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-25T08:18:10.289 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-25T08:18:10.345 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-25T08:18:10.346 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:18:10.346 INFO:teuthology.orchestra.run.trial169.stdout:Installed: 2026-02-25T08:18:10.346 INFO:teuthology.orchestra.run.trial169.stdout: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:18:10.346 INFO:teuthology.orchestra.run.trial169.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-25T08:18:10.346 INFO:teuthology.orchestra.run.trial169.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-25T08:18:10.346 INFO:teuthology.orchestra.run.trial169.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-25T08:18:10.346 INFO:teuthology.orchestra.run.trial169.stdout: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:18:10.346 INFO:teuthology.orchestra.run.trial169.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-25T08:18:10.346 INFO:teuthology.orchestra.run.trial169.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-25T08:18:10.347 INFO:teuthology.orchestra.run.trial169.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-25T08:18:10.347 INFO:teuthology.orchestra.run.trial169.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-25T08:18:10.347 INFO:teuthology.orchestra.run.trial169.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-25T08:18:10.347 INFO:teuthology.orchestra.run.trial169.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-25T08:18:10.347 INFO:teuthology.orchestra.run.trial169.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-25T08:18:10.347 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:18:10.347 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:18:10.386 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph 2026-02-25T08:18:10.696 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:09:02 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:18:10.783 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:18:10.786 INFO:teuthology.orchestra.run.trial169.stdout:======================================================================================= 2026-02-25T08:18:10.786 INFO:teuthology.orchestra.run.trial169.stdout: Package Arch Version Repository Size 2026-02-25T08:18:10.786 INFO:teuthology.orchestra.run.trial169.stdout:======================================================================================= 2026-02-25T08:18:10.786 INFO:teuthology.orchestra.run.trial169.stdout:Installing: 2026-02-25T08:18:10.786 INFO:teuthology.orchestra.run.trial169.stdout: ceph x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 6.5 k 2026-02-25T08:18:10.786 INFO:teuthology.orchestra.run.trial169.stdout:Installing dependencies: 2026-02-25T08:18:10.786 INFO:teuthology.orchestra.run.trial169.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-25T08:18:10.786 INFO:teuthology.orchestra.run.trial169.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 57 k 2026-02-25T08:18:10.786 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mds x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.0 M 2026-02-25T08:18:10.786 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mgr x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 937 k 2026-02-25T08:18:10.786 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 297 k 2026-02-25T08:18:10.786 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 4.9 M 2026-02-25T08:18:10.786 INFO:teuthology.orchestra.run.trial169.stdout: ceph-osd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 163 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: ceph-osd-classic x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 15 M 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: ceph-osd-crimson x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 88 M 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 17 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 1.0 M 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-25T08:18:10.787 INFO:teuthology.orchestra.run.trial169.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout:Installing weak dependencies: 2026-02-25T08:18:10.788 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 184 k 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 51 M 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 7.4 M 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 22 k 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mgr-rook noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 50 k 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout: ceph-volume noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 301 k 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout:Transaction Summary 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout:======================================================================================= 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout:Install 96 Packages 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout:Total download size: 226 M 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout:Installed size: 1.5 G 2026-02-25T08:18:10.789 INFO:teuthology.orchestra.run.trial169.stdout:Downloading Packages: 2026-02-25T08:18:11.619 INFO:teuthology.orchestra.run.trial169.stdout:(1/96): ceph-20.3.0-5338.g6aeb6975.el9.x86_64.r 22 kB/s | 6.5 kB 00:00 2026-02-25T08:18:12.908 INFO:teuthology.orchestra.run.trial169.stdout:(2/96): ceph-mgr-20.3.0-5338.g6aeb6975.el9.x86_ 592 kB/s | 937 kB 00:01 2026-02-25T08:18:13.540 INFO:teuthology.orchestra.run.trial169.stdout:(3/96): ceph-osd-20.3.0-5338.g6aeb6975.el9.x86_ 258 kB/s | 163 kB 00:00 2026-02-25T08:18:20.352 INFO:teuthology.orchestra.run.trial169.stdout:(4/96): ceph-mds-20.3.0-5338.g6aeb6975.el9.x86_ 229 kB/s | 2.0 MB 00:09 2026-02-25T08:18:26.597 INFO:teuthology.orchestra.run.trial169.stdout:(5/96): ceph-mon-20.3.0-5338.g6aeb6975.el9.x86_ 333 kB/s | 4.9 MB 00:14 2026-02-25T08:18:26.809 INFO:teuthology.orchestra.run.trial169.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5338.g6a 271 kB/s | 57 kB 00:00 2026-02-25T08:18:27.300 INFO:teuthology.orchestra.run.trial169.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5338.g6aeb6975. 374 kB/s | 184 kB 00:00 2026-02-25T08:19:29.620 INFO:teuthology.orchestra.run.trial169.stdout:(8/96): ceph-osd-classic-20.3.0-5338.g6aeb6975. 199 kB/s | 15 MB 01:16 2026-02-25T08:20:01.685 INFO:teuthology.orchestra.run.trial169.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 237 kB/s | 7.4 MB 00:32 2026-02-25T08:20:01.923 INFO:teuthology.orchestra.run.trial169.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5338.g6aeb69 94 kB/s | 22 kB 00:00 2026-02-25T08:20:03.198 INFO:teuthology.orchestra.run.trial169.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5338.g6ae 233 kB/s | 297 kB 00:01 2026-02-25T08:20:03.412 INFO:teuthology.orchestra.run.trial169.stdout:(12/96): ceph-mgr-rook-20.3.0-5338.g6aeb6975.el 236 kB/s | 50 kB 00:00 2026-02-25T08:20:03.649 INFO:teuthology.orchestra.run.trial169.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5338.g6a 73 kB/s | 17 kB 00:00 2026-02-25T08:20:04.671 INFO:teuthology.orchestra.run.trial169.stdout:(14/96): ceph-volume-20.3.0-5338.g6aeb6975.el9. 295 kB/s | 301 kB 00:01 2026-02-25T08:20:08.617 INFO:teuthology.orchestra.run.trial169.stdout:(15/96): cephadm-20.3.0-5338.g6aeb6975.el9.noar 264 kB/s | 1.0 MB 00:03 2026-02-25T08:20:08.733 INFO:teuthology.orchestra.run.trial169.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 3.0 MB/s | 351 kB 00:00 2026-02-25T08:20:08.755 INFO:teuthology.orchestra.run.trial169.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 1.8 MB/s | 40 kB 00:00 2026-02-25T08:20:08.777 INFO:teuthology.orchestra.run.trial169.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 3.2 MB/s | 72 kB 00:00 2026-02-25T08:20:08.821 INFO:teuthology.orchestra.run.trial169.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 18 MB/s | 794 kB 00:00 2026-02-25T08:20:08.846 INFO:teuthology.orchestra.run.trial169.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 7.4 MB/s | 184 kB 00:00 2026-02-25T08:20:08.870 INFO:teuthology.orchestra.run.trial169.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 10 MB/s | 253 kB 00:00 2026-02-25T08:20:08.938 INFO:teuthology.orchestra.run.trial169.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 18 MB/s | 1.2 MB 00:00 2026-02-25T08:20:08.962 INFO:teuthology.orchestra.run.trial169.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 4.5 MB/s | 106 kB 00:00 2026-02-25T08:20:08.985 INFO:teuthology.orchestra.run.trial169.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 5.6 MB/s | 135 kB 00:00 2026-02-25T08:20:09.009 INFO:teuthology.orchestra.run.trial169.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 6.2 MB/s | 150 kB 00:00 2026-02-25T08:20:09.033 INFO:teuthology.orchestra.run.trial169.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 5.1 MB/s | 126 kB 00:00 2026-02-25T08:20:09.058 INFO:teuthology.orchestra.run.trial169.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 8.8 MB/s | 218 kB 00:00 2026-02-25T08:20:09.082 INFO:teuthology.orchestra.run.trial169.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 7.3 MB/s | 182 kB 00:00 2026-02-25T08:20:09.106 INFO:teuthology.orchestra.run.trial169.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 11 MB/s | 266 kB 00:00 2026-02-25T08:20:09.126 INFO:teuthology.orchestra.run.trial169.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 1.4 MB/s | 30 kB 00:00 2026-02-25T08:20:09.153 INFO:teuthology.orchestra.run.trial169.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 111 MB/s | 3.0 MB 00:00 2026-02-25T08:20:09.157 INFO:teuthology.orchestra.run.trial169.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 3.6 MB/s | 15 kB 00:00 2026-02-25T08:20:09.163 INFO:teuthology.orchestra.run.trial169.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 40 MB/s | 246 kB 00:00 2026-02-25T08:20:09.169 INFO:teuthology.orchestra.run.trial169.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 31 MB/s | 188 kB 00:00 2026-02-25T08:20:09.174 INFO:teuthology.orchestra.run.trial169.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 9.5 MB/s | 42 kB 00:00 2026-02-25T08:20:09.195 INFO:teuthology.orchestra.run.trial169.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 258 MB/s | 5.3 MB 00:00 2026-02-25T08:20:09.219 INFO:teuthology.orchestra.run.trial169.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 247 MB/s | 6.0 MB 00:00 2026-02-25T08:20:09.226 INFO:teuthology.orchestra.run.trial169.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 34 MB/s | 244 kB 00:00 2026-02-25T08:20:09.233 INFO:teuthology.orchestra.run.trial169.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 36 MB/s | 249 kB 00:00 2026-02-25T08:20:09.238 INFO:teuthology.orchestra.run.trial169.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 9.3 MB/s | 48 kB 00:00 2026-02-25T08:20:09.244 INFO:teuthology.orchestra.run.trial169.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 30 MB/s | 177 kB 00:00 2026-02-25T08:20:09.253 INFO:teuthology.orchestra.run.trial169.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 130 MB/s | 1.2 MB 00:00 2026-02-25T08:20:09.258 INFO:teuthology.orchestra.run.trial169.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 7.0 MB/s | 35 kB 00:00 2026-02-25T08:20:09.281 INFO:teuthology.orchestra.run.trial169.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 266 MB/s | 6.1 MB 00:00 2026-02-25T08:20:09.289 INFO:teuthology.orchestra.run.trial169.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 56 MB/s | 442 kB 00:00 2026-02-25T08:20:09.295 INFO:teuthology.orchestra.run.trial169.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 14 MB/s | 77 kB 00:00 2026-02-25T08:20:09.301 INFO:teuthology.orchestra.run.trial169.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 45 MB/s | 267 kB 00:00 2026-02-25T08:20:09.307 INFO:teuthology.orchestra.run.trial169.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 24 MB/s | 157 kB 00:00 2026-02-25T08:20:09.314 INFO:teuthology.orchestra.run.trial169.stdout:(49/96): python3-pyasn1-modules-0.4.8-7.el9.noa 40 MB/s | 277 kB 00:00 2026-02-25T08:20:09.319 INFO:teuthology.orchestra.run.trial169.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 11 MB/s | 54 kB 00:00 2026-02-25T08:20:09.379 INFO:teuthology.orchestra.run.trial169.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 322 MB/s | 19 MB 00:00 2026-02-25T08:20:09.384 INFO:teuthology.orchestra.run.trial169.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 7.8 MB/s | 42 kB 00:00 2026-02-25T08:20:09.390 INFO:teuthology.orchestra.run.trial169.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 34 MB/s | 189 kB 00:00 2026-02-25T08:20:09.395 INFO:teuthology.orchestra.run.trial169.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 17 MB/s | 90 kB 00:00 2026-02-25T08:20:09.847 INFO:teuthology.orchestra.run.trial169.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 49 kB/s | 22 kB 00:00 2026-02-25T08:20:11.001 INFO:teuthology.orchestra.run.trial169.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 747 kB/s | 862 kB 00:01 2026-02-25T08:20:12.062 INFO:teuthology.orchestra.run.trial169.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 520 kB/s | 551 kB 00:01 2026-02-25T08:20:12.144 INFO:teuthology.orchestra.run.trial169.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 237 kB/s | 19 kB 00:00 2026-02-25T08:20:12.388 INFO:teuthology.orchestra.run.trial169.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 620 kB/s | 151 kB 00:00 2026-02-25T08:20:13.037 INFO:teuthology.orchestra.run.trial169.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 845 kB/s | 548 kB 00:00 2026-02-25T08:20:13.118 INFO:teuthology.orchestra.run.trial169.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 361 kB/s | 29 kB 00:00 2026-02-25T08:20:13.200 INFO:teuthology.orchestra.run.trial169.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 733 kB/s | 60 kB 00:00 2026-02-25T08:20:13.282 INFO:teuthology.orchestra.run.trial169.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 531 kB/s | 43 kB 00:00 2026-02-25T08:20:13.365 INFO:teuthology.orchestra.run.trial169.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 392 kB/s | 32 kB 00:00 2026-02-25T08:20:13.446 INFO:teuthology.orchestra.run.trial169.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 173 kB/s | 14 kB 00:00 2026-02-25T08:20:13.690 INFO:teuthology.orchestra.run.trial169.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 711 kB/s | 173 kB 00:00 2026-02-25T08:20:14.177 INFO:teuthology.orchestra.run.trial169.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 737 kB/s | 358 kB 00:00 2026-02-25T08:20:14.502 INFO:teuthology.orchestra.run.trial169.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 783 kB/s | 254 kB 00:00 2026-02-25T08:20:16.932 INFO:teuthology.orchestra.run.trial169.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 859 kB/s | 2.0 MB 00:02 2026-02-25T08:20:17.176 INFO:teuthology.orchestra.run.trial169.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 592 kB/s | 144 kB 00:00 2026-02-25T08:20:17.339 INFO:teuthology.orchestra.run.trial169.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 854 kB/s | 139 kB 00:00 2026-02-25T08:20:17.421 INFO:teuthology.orchestra.run.trial169.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 684 kB/s | 56 kB 00:00 2026-02-25T08:20:17.503 INFO:teuthology.orchestra.run.trial169.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 131 kB/s | 11 kB 00:00 2026-02-25T08:20:17.585 INFO:teuthology.orchestra.run.trial169.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 217 kB/s | 18 kB 00:00 2026-02-25T08:20:17.666 INFO:teuthology.orchestra.run.trial169.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 283 kB/s | 23 kB 00:00 2026-02-25T08:20:17.748 INFO:teuthology.orchestra.run.trial169.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 241 kB/s | 20 kB 00:00 2026-02-25T08:20:17.830 INFO:teuthology.orchestra.run.trial169.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 238 kB/s | 19 kB 00:00 2026-02-25T08:20:17.912 INFO:teuthology.orchestra.run.trial169.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 321 kB/s | 26 kB 00:00 2026-02-25T08:20:18.482 INFO:teuthology.orchestra.run.trial169.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2026-02-25T08:20:18.564 INFO:teuthology.orchestra.run.trial169.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 960 kB/s | 79 kB 00:00 2026-02-25T08:20:18.646 INFO:teuthology.orchestra.run.trial169.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 1.0 MB/s | 86 kB 00:00 2026-02-25T08:20:18.729 INFO:teuthology.orchestra.run.trial169.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 701 kB/s | 58 kB 00:00 2026-02-25T08:20:18.811 INFO:teuthology.orchestra.run.trial169.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 199 kB/s | 16 kB 00:00 2026-02-25T08:20:18.893 INFO:teuthology.orchestra.run.trial169.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2026-02-25T08:20:18.975 INFO:teuthology.orchestra.run.trial169.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 376 kB/s | 31 kB 00:00 2026-02-25T08:20:19.058 INFO:teuthology.orchestra.run.trial169.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2026-02-25T08:20:19.140 INFO:teuthology.orchestra.run.trial169.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 718 kB/s | 59 kB 00:00 2026-02-25T08:20:19.223 INFO:teuthology.orchestra.run.trial169.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 1.5 MB/s | 125 kB 00:00 2026-02-25T08:20:19.306 INFO:teuthology.orchestra.run.trial169.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 431 kB/s | 36 kB 00:00 2026-02-25T08:20:19.388 INFO:teuthology.orchestra.run.trial169.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 1.0 MB/s | 86 kB 00:00 2026-02-25T08:20:19.471 INFO:teuthology.orchestra.run.trial169.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2026-02-25T08:20:19.553 INFO:teuthology.orchestra.run.trial169.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 591 kB/s | 48 kB 00:00 2026-02-25T08:20:19.635 INFO:teuthology.orchestra.run.trial169.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 271 kB/s | 22 kB 00:00 2026-02-25T08:20:19.717 INFO:teuthology.orchestra.run.trial169.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 243 kB/s | 20 kB 00:00 2026-02-25T08:21:10.726 INFO:teuthology.orchestra.run.trial169.stdout:(95/96): ceph-mgr-dashboard-20.3.0-5338.g6aeb69 318 kB/s | 51 MB 02:43 2026-02-25T08:21:55.976 INFO:teuthology.orchestra.run.trial183.stdout:(12/12): ceph-test-20.3.0-5338.g6aeb6975.el9.x8 241 kB/s | 127 MB 08:58 2026-02-25T08:21:55.978 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:21:55.978 INFO:teuthology.orchestra.run.trial183.stdout:Total 250 kB/s | 132 MB 08:58 2026-02-25T08:21:56.047 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-25T08:21:56.059 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-25T08:21:56.059 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-25T08:21:56.491 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-25T08:21:56.491 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-25T08:21:56.951 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-25T08:21:56.960 INFO:teuthology.orchestra.run.trial183.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/12 2026-02-25T08:21:56.968 INFO:teuthology.orchestra.run.trial183.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/12 2026-02-25T08:21:56.973 INFO:teuthology.orchestra.run.trial183.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/12 2026-02-25T08:21:56.977 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/12 2026-02-25T08:21:56.980 INFO:teuthology.orchestra.run.trial183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/12 2026-02-25T08:21:56.984 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libubsan-11.5.0-14.el9.x86_64 6/12 2026-02-25T08:21:56.985 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:21:57.000 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libasan8-15.2.1-7.el9.x86_64 7/12 2026-02-25T08:21:57.003 INFO:teuthology.orchestra.run.trial183.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 8/12 2026-02-25T08:21:57.012 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 9/12 2026-02-25T08:21:57.021 INFO:teuthology.orchestra.run.trial183.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 10/12 2026-02-25T08:21:57.022 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:22:04.303 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 11/12 2026-02-25T08:22:04.322 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:22:05.121 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 12/12 2026-02-25T08:22:05.121 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/12 2026-02-25T08:22:05.121 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/12 2026-02-25T08:22:05.121 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/12 2026-02-25T08:22:05.121 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/12 2026-02-25T08:22:05.121 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/12 2026-02-25T08:22:05.121 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libasan8-15.2.1-7.el9.x86_64 6/12 2026-02-25T08:22:05.122 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libubsan-11.5.0-14.el9.x86_64 7/12 2026-02-25T08:22:05.122 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 8/12 2026-02-25T08:22:05.122 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 9/12 2026-02-25T08:22:05.122 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 10/12 2026-02-25T08:22:05.122 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 11/12 2026-02-25T08:22:05.181 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 12/12 2026-02-25T08:22:05.182 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:22:05.182 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-25T08:22:05.182 INFO:teuthology.orchestra.run.trial183.stdout: ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:05.182 INFO:teuthology.orchestra.run.trial183.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-25T08:22:05.182 INFO:teuthology.orchestra.run.trial183.stdout: libasan8-15.2.1-7.el9.x86_64 2026-02-25T08:22:05.182 INFO:teuthology.orchestra.run.trial183.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-25T08:22:05.182 INFO:teuthology.orchestra.run.trial183.stdout: libcephsqlite-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:22:05.182 INFO:teuthology.orchestra.run.trial183.stdout: libubsan-11.5.0-14.el9.x86_64 2026-02-25T08:22:05.182 INFO:teuthology.orchestra.run.trial183.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-25T08:22:05.182 INFO:teuthology.orchestra.run.trial183.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-25T08:22:05.183 INFO:teuthology.orchestra.run.trial183.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-25T08:22:05.183 INFO:teuthology.orchestra.run.trial183.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-25T08:22:05.183 INFO:teuthology.orchestra.run.trial183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-25T08:22:05.183 INFO:teuthology.orchestra.run.trial183.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-25T08:22:05.183 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:22:05.183 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:22:05.223 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph 2026-02-25T08:22:05.546 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:12:42 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:22:05.636 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:22:05.639 INFO:teuthology.orchestra.run.trial183.stdout:======================================================================================= 2026-02-25T08:22:05.639 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repository Size 2026-02-25T08:22:05.639 INFO:teuthology.orchestra.run.trial183.stdout:======================================================================================= 2026-02-25T08:22:05.639 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-25T08:22:05.639 INFO:teuthology.orchestra.run.trial183.stdout: ceph x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 6.5 k 2026-02-25T08:22:05.639 INFO:teuthology.orchestra.run.trial183.stdout:Installing dependencies: 2026-02-25T08:22:05.640 INFO:teuthology.orchestra.run.trial183.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-25T08:22:05.640 INFO:teuthology.orchestra.run.trial183.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 57 k 2026-02-25T08:22:05.640 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mds x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.0 M 2026-02-25T08:22:05.640 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 937 k 2026-02-25T08:22:05.640 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 297 k 2026-02-25T08:22:05.640 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mon x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 4.9 M 2026-02-25T08:22:05.640 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 163 k 2026-02-25T08:22:05.640 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-classic x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 15 M 2026-02-25T08:22:05.640 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-crimson x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 88 M 2026-02-25T08:22:05.640 INFO:teuthology.orchestra.run.trial183.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 17 k 2026-02-25T08:22:05.640 INFO:teuthology.orchestra.run.trial183.stdout: cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 1.0 M 2026-02-25T08:22:05.640 INFO:teuthology.orchestra.run.trial183.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-25T08:22:05.640 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-25T08:22:05.641 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-25T08:22:05.641 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-25T08:22:05.641 INFO:teuthology.orchestra.run.trial183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-25T08:22:05.641 INFO:teuthology.orchestra.run.trial183.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-25T08:22:05.641 INFO:teuthology.orchestra.run.trial183.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-25T08:22:05.641 INFO:teuthology.orchestra.run.trial183.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-25T08:22:05.641 INFO:teuthology.orchestra.run.trial183.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-25T08:22:05.641 INFO:teuthology.orchestra.run.trial183.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-25T08:22:05.641 INFO:teuthology.orchestra.run.trial183.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-25T08:22:05.641 INFO:teuthology.orchestra.run.trial183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-25T08:22:05.641 INFO:teuthology.orchestra.run.trial183.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-25T08:22:05.641 INFO:teuthology.orchestra.run.trial183.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-25T08:22:05.641 INFO:teuthology.orchestra.run.trial183.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-25T08:22:05.642 INFO:teuthology.orchestra.run.trial183.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-25T08:22:05.642 INFO:teuthology.orchestra.run.trial183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-25T08:22:05.642 INFO:teuthology.orchestra.run.trial183.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-25T08:22:05.642 INFO:teuthology.orchestra.run.trial183.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-25T08:22:05.642 INFO:teuthology.orchestra.run.trial183.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-25T08:22:05.642 INFO:teuthology.orchestra.run.trial183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-25T08:22:05.642 INFO:teuthology.orchestra.run.trial183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-25T08:22:05.642 INFO:teuthology.orchestra.run.trial183.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-25T08:22:05.642 INFO:teuthology.orchestra.run.trial183.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-25T08:22:05.642 INFO:teuthology.orchestra.run.trial183.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-25T08:22:05.642 INFO:teuthology.orchestra.run.trial183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-25T08:22:05.642 INFO:teuthology.orchestra.run.trial183.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-25T08:22:05.642 INFO:teuthology.orchestra.run.trial183.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-25T08:22:05.643 INFO:teuthology.orchestra.run.trial183.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-25T08:22:05.643 INFO:teuthology.orchestra.run.trial183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-25T08:22:05.643 INFO:teuthology.orchestra.run.trial183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-25T08:22:05.643 INFO:teuthology.orchestra.run.trial183.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-25T08:22:05.643 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-25T08:22:05.643 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-25T08:22:05.643 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-25T08:22:05.643 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-25T08:22:05.643 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-25T08:22:05.643 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-25T08:22:05.643 INFO:teuthology.orchestra.run.trial183.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-25T08:22:05.643 INFO:teuthology.orchestra.run.trial183.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-25T08:22:05.643 INFO:teuthology.orchestra.run.trial183.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-25T08:22:05.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-25T08:22:05.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-25T08:22:05.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-25T08:22:05.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-25T08:22:05.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-25T08:22:05.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-25T08:22:05.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-25T08:22:05.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-25T08:22:05.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-25T08:22:05.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-25T08:22:05.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-25T08:22:05.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-25T08:22:05.644 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-25T08:22:05.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyasn1 noarch 0.4.8-7.el9 appstream 157 k 2026-02-25T08:22:05.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyasn1-modules noarch 0.4.8-7.el9 appstream 277 k 2026-02-25T08:22:05.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-25T08:22:05.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-25T08:22:05.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-25T08:22:05.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-25T08:22:05.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-25T08:22:05.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-25T08:22:05.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-25T08:22:05.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-25T08:22:05.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-25T08:22:05.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-25T08:22:05.645 INFO:teuthology.orchestra.run.trial183.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-25T08:22:05.646 INFO:teuthology.orchestra.run.trial183.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-25T08:22:05.646 INFO:teuthology.orchestra.run.trial183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-25T08:22:05.646 INFO:teuthology.orchestra.run.trial183.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-25T08:22:05.646 INFO:teuthology.orchestra.run.trial183.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-25T08:22:05.646 INFO:teuthology.orchestra.run.trial183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-25T08:22:05.646 INFO:teuthology.orchestra.run.trial183.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-25T08:22:05.646 INFO:teuthology.orchestra.run.trial183.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-25T08:22:05.646 INFO:teuthology.orchestra.run.trial183.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-25T08:22:05.646 INFO:teuthology.orchestra.run.trial183.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-25T08:22:05.646 INFO:teuthology.orchestra.run.trial183.stdout:Installing weak dependencies: 2026-02-25T08:22:05.646 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 184 k 2026-02-25T08:22:05.646 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 51 M 2026-02-25T08:22:05.646 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 7.4 M 2026-02-25T08:22:05.647 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 22 k 2026-02-25T08:22:05.647 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-rook noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 50 k 2026-02-25T08:22:05.647 INFO:teuthology.orchestra.run.trial183.stdout: ceph-volume noarch 2:20.3.0-5338.g6aeb6975.el9 ceph-noarch 301 k 2026-02-25T08:22:05.647 INFO:teuthology.orchestra.run.trial183.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-25T08:22:05.647 INFO:teuthology.orchestra.run.trial183.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-25T08:22:05.647 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:22:05.647 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-25T08:22:05.647 INFO:teuthology.orchestra.run.trial183.stdout:======================================================================================= 2026-02-25T08:22:05.647 INFO:teuthology.orchestra.run.trial183.stdout:Install 96 Packages 2026-02-25T08:22:05.647 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:22:05.648 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 226 M 2026-02-25T08:22:05.648 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 1.5 G 2026-02-25T08:22:05.648 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-25T08:22:06.219 INFO:teuthology.orchestra.run.trial183.stdout:(1/96): ceph-20.3.0-5338.g6aeb6975.el9.x86_64.r 23 kB/s | 6.5 kB 00:00 2026-02-25T08:22:08.443 INFO:teuthology.orchestra.run.trial183.stdout:(2/96): ceph-mgr-20.3.0-5338.g6aeb6975.el9.x86_ 374 kB/s | 937 kB 00:02 2026-02-25T08:22:08.791 INFO:teuthology.orchestra.run.trial183.stdout:(3/96): ceph-osd-20.3.0-5338.g6aeb6975.el9.x86_ 468 kB/s | 163 kB 00:00 2026-02-25T08:22:12.834 INFO:teuthology.orchestra.run.trial183.stdout:(4/96): ceph-mds-20.3.0-5338.g6aeb6975.el9.x86_ 300 kB/s | 2.0 MB 00:06 2026-02-25T08:22:24.931 INFO:teuthology.orchestra.run.trial183.stdout:(5/96): ceph-mon-20.3.0-5338.g6aeb6975.el9.x86_ 266 kB/s | 4.9 MB 00:18 2026-02-25T08:22:25.207 INFO:teuthology.orchestra.run.trial183.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5338.g6a 207 kB/s | 57 kB 00:00 2026-02-25T08:22:26.042 INFO:teuthology.orchestra.run.trial183.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5338.g6aeb6975. 220 kB/s | 184 kB 00:00 2026-02-25T08:22:41.740 INFO:teuthology.orchestra.run.trial183.stdout:(8/96): ceph-osd-classic-20.3.0-5338.g6aeb6975. 459 kB/s | 15 MB 00:32 2026-02-25T08:22:57.313 INFO:teuthology.orchestra.run.trial183.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 487 kB/s | 7.4 MB 00:15 2026-02-25T08:22:57.384 INFO:teuthology.orchestra.run.trial183.stdout:(10/96): ceph-mgr-k8sevents-20.3.0-5338.g6aeb69 314 kB/s | 22 kB 00:00 2026-02-25T08:22:58.109 INFO:teuthology.orchestra.run.trial183.stdout:(11/96): ceph-mgr-modules-core-20.3.0-5338.g6ae 409 kB/s | 297 kB 00:00 2026-02-25T08:22:58.311 INFO:teuthology.orchestra.run.trial183.stdout:(12/96): ceph-mgr-rook-20.3.0-5338.g6aeb6975.el 249 kB/s | 50 kB 00:00 2026-02-25T08:22:58.381 INFO:teuthology.orchestra.run.trial183.stdout:(13/96): ceph-prometheus-alerts-20.3.0-5338.g6a 248 kB/s | 17 kB 00:00 2026-02-25T08:22:59.289 INFO:teuthology.orchestra.run.trial183.stdout:(14/96): ceph-volume-20.3.0-5338.g6aeb6975.el9. 332 kB/s | 301 kB 00:00 2026-02-25T08:23:02.007 INFO:teuthology.orchestra.run.trial183.stdout:(15/96): cephadm-20.3.0-5338.g6aeb6975.el9.noar 383 kB/s | 1.0 MB 00:02 2026-02-25T08:23:02.847 INFO:teuthology.orchestra.run.trial183.stdout:(16/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 418 kB/s | 351 kB 00:00 2026-02-25T08:23:02.933 INFO:teuthology.orchestra.run.trial183.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 468 kB/s | 40 kB 00:00 2026-02-25T08:23:03.101 INFO:teuthology.orchestra.run.trial183.stdout:(18/96): libconfig-1.7.2-9.el9.x86_64.rpm 429 kB/s | 72 kB 00:00 2026-02-25T08:23:04.089 INFO:teuthology.orchestra.run.trial183.stdout:(19/96): libgfortran-11.5.0-14.el9.x86_64.rpm 805 kB/s | 794 kB 00:00 2026-02-25T08:23:04.339 INFO:teuthology.orchestra.run.trial183.stdout:(20/96): libquadmath-11.5.0-14.el9.x86_64.rpm 737 kB/s | 184 kB 00:00 2026-02-25T08:23:04.672 INFO:teuthology.orchestra.run.trial183.stdout:(21/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 761 kB/s | 253 kB 00:00 2026-02-25T08:23:06.069 INFO:teuthology.orchestra.run.trial183.stdout:(22/96): python3-cryptography-36.0.1-5.el9.x86_ 913 kB/s | 1.2 MB 00:01 2026-02-25T08:23:06.157 INFO:teuthology.orchestra.run.trial183.stdout:(23/96): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2026-02-25T08:23:06.244 INFO:teuthology.orchestra.run.trial183.stdout:(24/96): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 135 kB 00:00 2026-02-25T08:23:06.332 INFO:teuthology.orchestra.run.trial183.stdout:(25/96): python3-pyparsing-2.4.7-9.el9.noarch.r 1.7 MB/s | 150 kB 00:00 2026-02-25T08:23:06.419 INFO:teuthology.orchestra.run.trial183.stdout:(26/96): python3-requests-2.25.1-10.el9.noarch. 1.4 MB/s | 126 kB 00:00 2026-02-25T08:23:06.507 INFO:teuthology.orchestra.run.trial183.stdout:(27/96): python3-urllib3-1.26.5-7.el9.noarch.rp 2.4 MB/s | 218 kB 00:00 2026-02-25T08:23:06.594 INFO:teuthology.orchestra.run.trial183.stdout:(28/96): unzip-6.0-59.el9.x86_64.rpm 2.0 MB/s | 182 kB 00:00 2026-02-25T08:23:06.682 INFO:teuthology.orchestra.run.trial183.stdout:(29/96): zip-3.0-35.el9.x86_64.rpm 3.0 MB/s | 266 kB 00:00 2026-02-25T08:23:06.725 INFO:teuthology.orchestra.run.trial183.stdout:(30/96): flexiblas-3.0.4-9.el9.x86_64.rpm 694 kB/s | 30 kB 00:00 2026-02-25T08:23:06.874 INFO:teuthology.orchestra.run.trial183.stdout:(31/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 20 MB/s | 3.0 MB 00:00 2026-02-25T08:23:06.896 INFO:teuthology.orchestra.run.trial183.stdout:(32/96): flexiblas-openblas-openmp-3.0.4-9.el9. 690 kB/s | 15 kB 00:00 2026-02-25T08:23:06.919 INFO:teuthology.orchestra.run.trial183.stdout:(33/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 10 MB/s | 246 kB 00:00 2026-02-25T08:23:06.943 INFO:teuthology.orchestra.run.trial183.stdout:(34/96): lua-5.4.4-4.el9.x86_64.rpm 7.9 MB/s | 188 kB 00:00 2026-02-25T08:23:06.965 INFO:teuthology.orchestra.run.trial183.stdout:(35/96): openblas-0.3.29-1.el9.x86_64.rpm 1.9 MB/s | 42 kB 00:00 2026-02-25T08:23:07.137 INFO:teuthology.orchestra.run.trial183.stdout:(36/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 31 MB/s | 5.3 MB 00:00 2026-02-25T08:23:07.313 INFO:teuthology.orchestra.run.trial183.stdout:(37/96): python3-babel-2.9.1-2.el9.noarch.rpm 34 MB/s | 6.0 MB 00:00 2026-02-25T08:23:07.338 INFO:teuthology.orchestra.run.trial183.stdout:(38/96): python3-devel-3.9.25-3.el9.x86_64.rpm 9.7 MB/s | 244 kB 00:00 2026-02-25T08:23:07.362 INFO:teuthology.orchestra.run.trial183.stdout:(39/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 9.9 MB/s | 249 kB 00:00 2026-02-25T08:23:07.385 INFO:teuthology.orchestra.run.trial183.stdout:(40/96): python3-jmespath-1.0.1-1.el9.noarch.rp 2.0 MB/s | 48 kB 00:00 2026-02-25T08:23:07.409 INFO:teuthology.orchestra.run.trial183.stdout:(41/96): python3-libstoragemgmt-1.10.1-1.el9.x8 7.4 MB/s | 177 kB 00:00 2026-02-25T08:23:07.456 INFO:teuthology.orchestra.run.trial183.stdout:(42/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 26 MB/s | 1.2 MB 00:00 2026-02-25T08:23:07.478 INFO:teuthology.orchestra.run.trial183.stdout:(43/96): python3-markupsafe-1.1.1-12.el9.x86_64 1.5 MB/s | 35 kB 00:00 2026-02-25T08:23:07.672 INFO:teuthology.orchestra.run.trial183.stdout:(44/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 32 MB/s | 6.1 MB 00:00 2026-02-25T08:23:07.697 INFO:teuthology.orchestra.run.trial183.stdout:(45/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 17 MB/s | 442 kB 00:00 2026-02-25T08:23:07.720 INFO:teuthology.orchestra.run.trial183.stdout:(46/96): python3-packaging-20.9-5.el9.noarch.rp 3.3 MB/s | 77 kB 00:00 2026-02-25T08:23:07.743 INFO:teuthology.orchestra.run.trial183.stdout:(47/96): python3-protobuf-3.14.0-17.el9.noarch. 11 MB/s | 267 kB 00:00 2026-02-25T08:23:07.767 INFO:teuthology.orchestra.run.trial183.stdout:(48/96): python3-pyasn1-0.4.8-7.el9.noarch.rpm 6.4 MB/s | 157 kB 00:00 2026-02-25T08:23:07.791 INFO:teuthology.orchestra.run.trial183.stdout:(49/96): python3-pyasn1-modules-0.4.8-7.el9.noa 11 MB/s | 277 kB 00:00 2026-02-25T08:23:07.814 INFO:teuthology.orchestra.run.trial183.stdout:(50/96): python3-requests-oauthlib-1.3.0-12.el9 2.3 MB/s | 54 kB 00:00 2026-02-25T08:23:08.353 INFO:teuthology.orchestra.run.trial183.stdout:(51/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 36 MB/s | 19 MB 00:00 2026-02-25T08:23:08.375 INFO:teuthology.orchestra.run.trial183.stdout:(52/96): python3-toml-0.10.2-6.el9.noarch.rpm 1.8 MB/s | 42 kB 00:00 2026-02-25T08:23:08.398 INFO:teuthology.orchestra.run.trial183.stdout:(53/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 8.1 MB/s | 189 kB 00:00 2026-02-25T08:23:08.421 INFO:teuthology.orchestra.run.trial183.stdout:(54/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 3.9 MB/s | 90 kB 00:00 2026-02-25T08:23:08.442 INFO:teuthology.orchestra.run.trial183.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 22 kB 00:00 2026-02-25T08:23:08.486 INFO:teuthology.orchestra.run.trial183.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 19 MB/s | 862 kB 00:00 2026-02-25T08:23:08.668 INFO:teuthology.orchestra.run.trial183.stdout:(57/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 3.0 MB/s | 551 kB 00:00 2026-02-25T08:23:08.693 INFO:teuthology.orchestra.run.trial183.stdout:(58/96): grpc-data-1.46.7-10.el9.noarch.rpm 794 kB/s | 19 kB 00:00 2026-02-25T08:23:08.720 INFO:teuthology.orchestra.run.trial183.stdout:(59/96): luarocks-3.9.2-5.el9.noarch.rpm 5.4 MB/s | 151 kB 00:00 2026-02-25T08:23:08.760 INFO:teuthology.orchestra.run.trial183.stdout:(60/96): python3-asyncssh-2.13.2-5.el9.noarch.r 13 MB/s | 548 kB 00:00 2026-02-25T08:23:08.785 INFO:teuthology.orchestra.run.trial183.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 1.2 MB/s | 29 kB 00:00 2026-02-25T08:23:08.811 INFO:teuthology.orchestra.run.trial183.stdout:(62/96): python3-backports-tarfile-1.2.0-1.el9. 2.3 MB/s | 60 kB 00:00 2026-02-25T08:23:08.836 INFO:teuthology.orchestra.run.trial183.stdout:(63/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2026-02-25T08:23:08.861 INFO:teuthology.orchestra.run.trial183.stdout:(64/96): python3-cachetools-4.2.4-1.el9.noarch. 1.3 MB/s | 32 kB 00:00 2026-02-25T08:23:08.885 INFO:teuthology.orchestra.run.trial183.stdout:(65/96): python3-certifi-2023.05.07-4.el9.noarc 576 kB/s | 14 kB 00:00 2026-02-25T08:23:08.913 INFO:teuthology.orchestra.run.trial183.stdout:(66/96): python3-cheroot-10.0.1-4.el9.noarch.rp 6.1 MB/s | 173 kB 00:00 2026-02-25T08:23:08.946 INFO:teuthology.orchestra.run.trial183.stdout:(67/96): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2026-02-25T08:23:08.976 INFO:teuthology.orchestra.run.trial183.stdout:(68/96): python3-google-auth-2.45.0-1.el9.noarc 8.4 MB/s | 254 kB 00:00 2026-02-25T08:23:09.050 INFO:teuthology.orchestra.run.trial183.stdout:(69/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 27 MB/s | 2.0 MB 00:00 2026-02-25T08:23:09.077 INFO:teuthology.orchestra.run.trial183.stdout:(70/96): python3-grpcio-tools-1.46.7-10.el9.x86 5.3 MB/s | 144 kB 00:00 2026-02-25T08:23:09.104 INFO:teuthology.orchestra.run.trial183.stdout:(71/96): python3-influxdb-5.3.1-1.el9.noarch.rp 5.0 MB/s | 139 kB 00:00 2026-02-25T08:23:09.130 INFO:teuthology.orchestra.run.trial183.stdout:(72/96): python3-isodate-0.6.1-3.el9.noarch.rpm 2.2 MB/s | 56 kB 00:00 2026-02-25T08:23:09.154 INFO:teuthology.orchestra.run.trial183.stdout:(73/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 438 kB/s | 11 kB 00:00 2026-02-25T08:23:09.179 INFO:teuthology.orchestra.run.trial183.stdout:(74/96): python3-jaraco-classes-3.2.1-5.el9.noa 718 kB/s | 18 kB 00:00 2026-02-25T08:23:09.204 INFO:teuthology.orchestra.run.trial183.stdout:(75/96): python3-jaraco-collections-3.0.0-8.el9 924 kB/s | 23 kB 00:00 2026-02-25T08:23:09.229 INFO:teuthology.orchestra.run.trial183.stdout:(76/96): python3-jaraco-context-6.0.1-3.el9.noa 792 kB/s | 20 kB 00:00 2026-02-25T08:23:09.254 INFO:teuthology.orchestra.run.trial183.stdout:(77/96): python3-jaraco-functools-3.5.0-2.el9.n 783 kB/s | 19 kB 00:00 2026-02-25T08:23:09.279 INFO:teuthology.orchestra.run.trial183.stdout:(78/96): python3-jaraco-text-4.0.0-2.el9.noarch 1.0 MB/s | 26 kB 00:00 2026-02-25T08:23:09.313 INFO:teuthology.orchestra.run.trial183.stdout:(79/96): python3-kubernetes-26.1.0-3.el9.noarch 31 MB/s | 1.0 MB 00:00 2026-02-25T08:23:09.337 INFO:teuthology.orchestra.run.trial183.stdout:(80/96): python3-more-itertools-8.12.0-2.el9.no 3.2 MB/s | 79 kB 00:00 2026-02-25T08:23:09.361 INFO:teuthology.orchestra.run.trial183.stdout:(81/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 3.5 MB/s | 86 kB 00:00 2026-02-25T08:23:09.385 INFO:teuthology.orchestra.run.trial183.stdout:(82/96): python3-natsort-7.1.1-5.el9.noarch.rpm 2.4 MB/s | 58 kB 00:00 2026-02-25T08:23:09.408 INFO:teuthology.orchestra.run.trial183.stdout:(83/96): python3-portend-3.1.0-2.el9.noarch.rpm 700 kB/s | 16 kB 00:00 2026-02-25T08:23:09.433 INFO:teuthology.orchestra.run.trial183.stdout:(84/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.6 MB/s | 90 kB 00:00 2026-02-25T08:23:09.456 INFO:teuthology.orchestra.run.trial183.stdout:(85/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.3 MB/s | 31 kB 00:00 2026-02-25T08:23:09.482 INFO:teuthology.orchestra.run.trial183.stdout:(86/96): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2026-02-25T08:23:09.505 INFO:teuthology.orchestra.run.trial183.stdout:(87/96): python3-rsa-4.9-2.el9.noarch.rpm 2.4 MB/s | 59 kB 00:00 2026-02-25T08:23:09.530 INFO:teuthology.orchestra.run.trial183.stdout:(88/96): python3-saml-1.16.0-1.el9.noarch.rpm 5.0 MB/s | 125 kB 00:00 2026-02-25T08:23:09.554 INFO:teuthology.orchestra.run.trial183.stdout:(89/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.5 MB/s | 36 kB 00:00 2026-02-25T08:23:09.578 INFO:teuthology.orchestra.run.trial183.stdout:(90/96): python3-typing-extensions-4.15.0-1.el9 3.5 MB/s | 86 kB 00:00 2026-02-25T08:23:09.603 INFO:teuthology.orchestra.run.trial183.stdout:(91/96): python3-websocket-client-1.2.3-2.el9.n 3.6 MB/s | 90 kB 00:00 2026-02-25T08:23:09.627 INFO:teuthology.orchestra.run.trial183.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 2.0 MB/s | 48 kB 00:00 2026-02-25T08:23:09.651 INFO:teuthology.orchestra.run.trial183.stdout:(93/96): python3-xmltodict-0.12.0-15.el9.noarch 936 kB/s | 22 kB 00:00 2026-02-25T08:23:09.674 INFO:teuthology.orchestra.run.trial183.stdout:(94/96): python3-zc-lockfile-2.0-10.el9.noarch. 845 kB/s | 20 kB 00:00 2026-02-25T08:23:43.442 INFO:teuthology.orchestra.run.trial169.stdout:(96/96): ceph-osd-crimson-20.3.0-5338.g6aeb6975 278 kB/s | 88 MB 05:23 2026-02-25T08:23:43.445 INFO:teuthology.orchestra.run.trial169.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:23:43.445 INFO:teuthology.orchestra.run.trial169.stdout:Total 695 kB/s | 226 MB 05:32 2026-02-25T08:23:43.821 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction check 2026-02-25T08:23:43.859 INFO:teuthology.orchestra.run.trial169.stdout:Transaction check succeeded. 2026-02-25T08:23:43.860 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction test 2026-02-25T08:23:44.667 INFO:teuthology.orchestra.run.trial169.stdout:Transaction test succeeded. 2026-02-25T08:23:44.668 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction 2026-02-25T08:23:45.501 INFO:teuthology.orchestra.run.trial169.stdout: Preparing : 1/1 2026-02-25T08:23:45.503 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-25T08:23:45.507 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-25T08:23:45.528 INFO:teuthology.orchestra.run.trial169.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-25T08:23:45.561 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-25T08:23:45.568 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-25T08:23:45.570 INFO:teuthology.orchestra.run.trial169.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-25T08:23:45.575 INFO:teuthology.orchestra.run.trial169.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-25T08:23:45.598 INFO:teuthology.orchestra.run.trial169.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-25T08:23:45.611 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-25T08:23:45.615 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-25T08:23:45.622 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-25T08:23:45.624 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-25T08:23:45.635 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-25T08:23:45.641 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-25T08:23:45.665 INFO:teuthology.orchestra.run.trial169.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-25T08:23:45.668 INFO:teuthology.orchestra.run.trial169.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-25T08:23:45.718 INFO:teuthology.orchestra.run.trial169.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-25T08:23:45.731 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-25T08:23:45.734 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-25T08:23:45.738 INFO:teuthology.orchestra.run.trial169.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-25T08:23:45.744 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-25T08:23:45.748 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-25T08:23:45.751 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-25T08:23:45.766 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-25T08:23:45.771 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-25T08:23:45.776 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-25T08:23:45.787 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-25T08:23:45.796 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-25T08:23:45.803 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-25T08:23:45.822 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-25T08:23:45.833 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-25T08:23:45.849 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-25T08:23:45.854 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-25T08:23:45.862 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-25T08:23:45.902 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-25T08:23:46.228 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-25T08:23:46.238 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-25T08:23:46.242 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-25T08:23:46.248 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-25T08:23:46.250 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-25T08:23:46.277 INFO:teuthology.orchestra.run.trial169.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-25T08:23:46.322 INFO:teuthology.orchestra.run.trial169.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-25T08:23:46.331 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-25T08:23:46.336 INFO:teuthology.orchestra.run.trial169.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-25T08:23:46.340 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-25T08:23:46.345 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-25T08:23:46.349 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-25T08:23:46.356 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-25T08:23:46.359 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-25T08:23:46.387 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-25T08:23:46.397 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-25T08:23:46.403 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-25T08:23:46.410 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-25T08:23:46.444 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-25T08:23:46.671 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-25T08:23:46.695 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-25T08:23:46.698 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-25T08:23:46.758 INFO:teuthology.orchestra.run.trial169.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-25T08:23:46.759 INFO:teuthology.orchestra.run.trial169.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-25T08:23:46.781 INFO:teuthology.orchestra.run.trial169.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-25T08:23:47.108 INFO:teuthology.orchestra.run.trial169.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-25T08:23:47.182 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-25T08:23:47.856 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-25T08:23:47.873 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-25T08:23:47.883 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-25T08:23:47.889 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-25T08:23:47.903 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-25T08:23:47.919 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-25T08:23:47.994 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-25T08:23:48.003 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-25T08:23:48.027 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-25T08:23:48.056 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-25T08:23:48.107 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-25T08:23:48.113 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-25T08:23:48.117 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-25T08:23:48.118 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-25T08:23:48.136 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:23:48.136 INFO:teuthology.orchestra.run.trial169.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-25T08:23:48.136 INFO:teuthology.orchestra.run.trial169.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-25T08:23:48.137 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:23:48.145 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:23:48.170 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:23:48.171 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-25T08:23:48.171 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:23:48.207 INFO:teuthology.orchestra.run.trial169.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-25T08:23:48.793 INFO:teuthology.orchestra.run.trial169.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-25T08:23:54.199 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 80/96 2026-02-25T08:23:54.263 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 81/96 2026-02-25T08:23:54.265 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:23:54.278 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:23:54.278 INFO:teuthology.orchestra.run.trial169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:23:54.279 INFO:teuthology.orchestra.run.trial169.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-25T08:23:54.279 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:23:54.289 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:23:54.311 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:23:54.311 INFO:teuthology.orchestra.run.trial169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:23:54.311 INFO:teuthology.orchestra.run.trial169.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-25T08:23:54.311 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:23:54.311 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:23:54.312 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:23:54.369 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:23:54.371 INFO:teuthology.orchestra.run.trial169.stdout: Installing : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:23:54.380 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 85/96 2026-02-25T08:23:54.404 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 86/96 2026-02-25T08:23:54.406 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:23:57.313 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:23:57.320 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:23:57.776 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:23:57.777 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:23:57.789 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:23:57.789 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:23:57.851 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:23:57.892 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 91/96 2026-02-25T08:23:57.893 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:23:57.914 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:23:57.914 INFO:teuthology.orchestra.run.trial169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:23:57.914 INFO:teuthology.orchestra.run.trial169.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-25T08:23:57.914 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:23:57.914 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:23:57.915 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:23:57.926 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:23:58.140 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:23:58.141 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:23:58.162 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:23:58.162 INFO:teuthology.orchestra.run.trial169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:23:58.162 INFO:teuthology.orchestra.run.trial169.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-25T08:23:58.162 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:23:58.162 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:23:58.163 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:23:58.245 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:23:58.265 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:23:58.265 INFO:teuthology.orchestra.run.trial169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:23:58.265 INFO:teuthology.orchestra.run.trial169.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-25T08:23:58.265 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:23:58.265 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:23:58.265 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:23:58.266 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:23:58.271 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:23:58.295 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:23:59.662 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:23:59.662 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/96 2026-02-25T08:23:59.662 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/96 2026-02-25T08:23:59.662 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/96 2026-02-25T08:23:59.663 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/96 2026-02-25T08:23:59.663 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 5/96 2026-02-25T08:23:59.663 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/96 2026-02-25T08:23:59.663 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 7/96 2026-02-25T08:23:59.663 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 8/96 2026-02-25T08:23:59.663 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 9/96 2026-02-25T08:23:59.663 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 10/96 2026-02-25T08:23:59.663 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 11/96 2026-02-25T08:23:59.663 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 12/96 2026-02-25T08:23:59.663 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 13/96 2026-02-25T08:23:59.663 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 14/96 2026-02-25T08:23:59.663 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 15/96 2026-02-25T08:23:59.664 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 16/96 2026-02-25T08:23:59.664 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 17/96 2026-02-25T08:23:59.664 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-25T08:23:59.664 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-25T08:23:59.664 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-25T08:23:59.664 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-25T08:23:59.664 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-25T08:23:59.664 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-25T08:23:59.664 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-25T08:23:59.664 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-25T08:23:59.664 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-25T08:23:59.665 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-25T08:23:59.665 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-25T08:23:59.665 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-25T08:23:59.665 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-25T08:23:59.665 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-25T08:23:59.665 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-25T08:23:59.665 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-25T08:23:59.665 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-25T08:23:59.665 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-25T08:23:59.665 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-25T08:23:59.666 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-25T08:23:59.666 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-25T08:23:59.666 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-25T08:23:59.666 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-25T08:23:59.666 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-25T08:23:59.666 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-25T08:23:59.666 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-25T08:23:59.666 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-25T08:23:59.666 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-25T08:23:59.666 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-25T08:23:59.666 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-25T08:23:59.666 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-25T08:23:59.667 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-25T08:23:59.667 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-25T08:23:59.667 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-25T08:23:59.667 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-25T08:23:59.667 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-25T08:23:59.667 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-25T08:23:59.667 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-25T08:23:59.667 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-25T08:23:59.667 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-25T08:23:59.667 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-25T08:23:59.667 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-25T08:23:59.667 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-25T08:23:59.667 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-25T08:23:59.668 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-25T08:23:59.668 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-25T08:23:59.668 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-25T08:23:59.668 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-25T08:23:59.668 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-25T08:23:59.668 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-25T08:23:59.668 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-25T08:23:59.668 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-25T08:23:59.668 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-25T08:23:59.668 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-25T08:23:59.668 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-25T08:23:59.668 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-25T08:23:59.668 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-25T08:23:59.668 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-25T08:23:59.669 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-25T08:23:59.669 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-25T08:23:59.669 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-25T08:23:59.669 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-25T08:23:59.669 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-25T08:23:59.669 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-25T08:23:59.669 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-25T08:23:59.669 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-25T08:23:59.669 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-25T08:23:59.669 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-25T08:23:59.669 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-25T08:23:59.669 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-25T08:23:59.670 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-25T08:23:59.670 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-25T08:23:59.670 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-25T08:23:59.670 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-25T08:23:59.670 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-25T08:23:59.670 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-25T08:23:59.670 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-25T08:23:59.670 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-25T08:23:59.730 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-25T08:23:59.730 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:23:59.730 INFO:teuthology.orchestra.run.trial169.stdout:Installed: 2026-02-25T08:23:59.730 INFO:teuthology.orchestra.run.trial169.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-25T08:23:59.730 INFO:teuthology.orchestra.run.trial169.stdout: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:23:59.730 INFO:teuthology.orchestra.run.trial169.stdout: ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:23:59.730 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:23:59.730 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:23:59.730 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:23:59.731 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:23:59.731 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:23:59.731 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:23:59.731 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:23:59.731 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:23:59.731 INFO:teuthology.orchestra.run.trial169.stdout: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:23:59.731 INFO:teuthology.orchestra.run.trial169.stdout: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:23:59.731 INFO:teuthology.orchestra.run.trial169.stdout: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:23:59.731 INFO:teuthology.orchestra.run.trial169.stdout: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:23:59.731 INFO:teuthology.orchestra.run.trial169.stdout: ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:23:59.732 INFO:teuthology.orchestra.run.trial169.stdout: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:23:59.732 INFO:teuthology.orchestra.run.trial169.stdout: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:23:59.732 INFO:teuthology.orchestra.run.trial169.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-25T08:23:59.732 INFO:teuthology.orchestra.run.trial169.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-25T08:23:59.732 INFO:teuthology.orchestra.run.trial169.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-25T08:23:59.732 INFO:teuthology.orchestra.run.trial169.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-25T08:23:59.732 INFO:teuthology.orchestra.run.trial169.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-25T08:23:59.732 INFO:teuthology.orchestra.run.trial169.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-25T08:23:59.732 INFO:teuthology.orchestra.run.trial169.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-25T08:23:59.732 INFO:teuthology.orchestra.run.trial169.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-25T08:23:59.732 INFO:teuthology.orchestra.run.trial169.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-25T08:23:59.732 INFO:teuthology.orchestra.run.trial169.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:23:59.732 INFO:teuthology.orchestra.run.trial169.stdout: lua-5.4.4-4.el9.x86_64 2026-02-25T08:23:59.733 INFO:teuthology.orchestra.run.trial169.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-25T08:23:59.733 INFO:teuthology.orchestra.run.trial169.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-25T08:23:59.733 INFO:teuthology.orchestra.run.trial169.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-25T08:23:59.733 INFO:teuthology.orchestra.run.trial169.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-25T08:23:59.733 INFO:teuthology.orchestra.run.trial169.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-25T08:23:59.733 INFO:teuthology.orchestra.run.trial169.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-25T08:23:59.733 INFO:teuthology.orchestra.run.trial169.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-25T08:23:59.733 INFO:teuthology.orchestra.run.trial169.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-25T08:23:59.733 INFO:teuthology.orchestra.run.trial169.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-25T08:23:59.733 INFO:teuthology.orchestra.run.trial169.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-25T08:23:59.733 INFO:teuthology.orchestra.run.trial169.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-25T08:23:59.733 INFO:teuthology.orchestra.run.trial169.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-25T08:23:59.734 INFO:teuthology.orchestra.run.trial169.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-25T08:23:59.734 INFO:teuthology.orchestra.run.trial169.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-25T08:23:59.734 INFO:teuthology.orchestra.run.trial169.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-25T08:23:59.734 INFO:teuthology.orchestra.run.trial169.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-25T08:23:59.734 INFO:teuthology.orchestra.run.trial169.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-25T08:23:59.734 INFO:teuthology.orchestra.run.trial169.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-25T08:23:59.734 INFO:teuthology.orchestra.run.trial169.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-25T08:23:59.734 INFO:teuthology.orchestra.run.trial169.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-25T08:23:59.734 INFO:teuthology.orchestra.run.trial169.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-25T08:23:59.734 INFO:teuthology.orchestra.run.trial169.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-25T08:23:59.734 INFO:teuthology.orchestra.run.trial169.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-25T08:23:59.734 INFO:teuthology.orchestra.run.trial169.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-25T08:23:59.735 INFO:teuthology.orchestra.run.trial169.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-25T08:23:59.735 INFO:teuthology.orchestra.run.trial169.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-25T08:23:59.735 INFO:teuthology.orchestra.run.trial169.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-25T08:23:59.735 INFO:teuthology.orchestra.run.trial169.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-25T08:23:59.735 INFO:teuthology.orchestra.run.trial169.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-25T08:23:59.735 INFO:teuthology.orchestra.run.trial169.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-25T08:23:59.735 INFO:teuthology.orchestra.run.trial169.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-25T08:23:59.735 INFO:teuthology.orchestra.run.trial169.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:23:59.735 INFO:teuthology.orchestra.run.trial169.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-25T08:23:59.735 INFO:teuthology.orchestra.run.trial169.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-25T08:23:59.735 INFO:teuthology.orchestra.run.trial169.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-25T08:23:59.735 INFO:teuthology.orchestra.run.trial169.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-25T08:23:59.735 INFO:teuthology.orchestra.run.trial169.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-25T08:23:59.736 INFO:teuthology.orchestra.run.trial169.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-25T08:23:59.736 INFO:teuthology.orchestra.run.trial169.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-25T08:23:59.736 INFO:teuthology.orchestra.run.trial169.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-25T08:23:59.736 INFO:teuthology.orchestra.run.trial169.stdout: python3-ply-3.11-14.el9.noarch 2026-02-25T08:23:59.736 INFO:teuthology.orchestra.run.trial169.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-25T08:23:59.736 INFO:teuthology.orchestra.run.trial169.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-25T08:23:59.736 INFO:teuthology.orchestra.run.trial169.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-25T08:23:59.736 INFO:teuthology.orchestra.run.trial169.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-25T08:23:59.736 INFO:teuthology.orchestra.run.trial169.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-25T08:23:59.736 INFO:teuthology.orchestra.run.trial169.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-25T08:23:59.736 INFO:teuthology.orchestra.run.trial169.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-25T08:23:59.736 INFO:teuthology.orchestra.run.trial169.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-25T08:23:59.736 INFO:teuthology.orchestra.run.trial169.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-25T08:23:59.737 INFO:teuthology.orchestra.run.trial169.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-25T08:23:59.737 INFO:teuthology.orchestra.run.trial169.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-25T08:23:59.737 INFO:teuthology.orchestra.run.trial169.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-25T08:23:59.737 INFO:teuthology.orchestra.run.trial169.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-25T08:23:59.737 INFO:teuthology.orchestra.run.trial169.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-25T08:23:59.737 INFO:teuthology.orchestra.run.trial169.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-25T08:23:59.737 INFO:teuthology.orchestra.run.trial169.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-25T08:23:59.737 INFO:teuthology.orchestra.run.trial169.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-25T08:23:59.737 INFO:teuthology.orchestra.run.trial169.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-25T08:23:59.737 INFO:teuthology.orchestra.run.trial169.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-25T08:23:59.737 INFO:teuthology.orchestra.run.trial169.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-25T08:23:59.737 INFO:teuthology.orchestra.run.trial169.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-25T08:23:59.737 INFO:teuthology.orchestra.run.trial169.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-25T08:23:59.738 INFO:teuthology.orchestra.run.trial169.stdout: unzip-6.0-59.el9.x86_64 2026-02-25T08:23:59.738 INFO:teuthology.orchestra.run.trial169.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-25T08:23:59.738 INFO:teuthology.orchestra.run.trial169.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-25T08:23:59.738 INFO:teuthology.orchestra.run.trial169.stdout: zip-3.0-35.el9.x86_64 2026-02-25T08:23:59.738 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:23:59.738 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:23:59.804 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-base 2026-02-25T08:24:00.128 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:14:52 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:00.193 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:24:00.212 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:00.213 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:00.213 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:00.236 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install cephadm 2026-02-25T08:24:00.557 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:14:52 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:00.623 INFO:teuthology.orchestra.run.trial169.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:24:00.642 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:00.642 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:00.642 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:00.664 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:24:00.984 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:14:52 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:01.068 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:01.069 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:01.069 INFO:teuthology.orchestra.run.trial169.stdout: Package Arch Version Repo Size 2026-02-25T08:24:01.069 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:01.070 INFO:teuthology.orchestra.run.trial169.stdout:Installing: 2026-02-25T08:24:01.070 INFO:teuthology.orchestra.run.trial169.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 149 k 2026-02-25T08:24:01.070 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:01.070 INFO:teuthology.orchestra.run.trial169.stdout:Transaction Summary 2026-02-25T08:24:01.070 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:01.070 INFO:teuthology.orchestra.run.trial169.stdout:Install 1 Package 2026-02-25T08:24:01.070 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:01.070 INFO:teuthology.orchestra.run.trial169.stdout:Total download size: 149 k 2026-02-25T08:24:01.070 INFO:teuthology.orchestra.run.trial169.stdout:Installed size: 455 k 2026-02-25T08:24:01.071 INFO:teuthology.orchestra.run.trial169.stdout:Downloading Packages: 2026-02-25T08:24:01.640 INFO:teuthology.orchestra.run.trial169.stdout:ceph-immutable-object-cache-20.3.0-5338.g6aeb69 262 kB/s | 149 kB 00:00 2026-02-25T08:24:01.641 INFO:teuthology.orchestra.run.trial169.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:24:01.641 INFO:teuthology.orchestra.run.trial169.stdout:Total 261 kB/s | 149 kB 00:00 2026-02-25T08:24:01.641 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction check 2026-02-25T08:24:01.646 INFO:teuthology.orchestra.run.trial169.stdout:Transaction check succeeded. 2026-02-25T08:24:01.646 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction test 2026-02-25T08:24:01.677 INFO:teuthology.orchestra.run.trial169.stdout:Transaction test succeeded. 2026-02-25T08:24:01.677 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction 2026-02-25T08:24:01.733 INFO:teuthology.orchestra.run.trial169.stdout: Preparing : 1/1 2026-02-25T08:24:01.735 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:24:01.757 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:24:01.758 INFO:teuthology.orchestra.run.trial169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:24:01.758 INFO:teuthology.orchestra.run.trial169.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-25T08:24:01.758 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:02.056 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:24:02.056 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:02.056 INFO:teuthology.orchestra.run.trial169.stdout:Installed: 2026-02-25T08:24:02.056 INFO:teuthology.orchestra.run.trial169.stdout: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:24:02.056 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:02.057 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:02.086 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-mgr 2026-02-25T08:24:02.413 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:14:54 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:02.479 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:24:02.498 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:02.499 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:02.499 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:02.521 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:24:02.833 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:14:54 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:02.897 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:24:02.916 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:02.917 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:02.917 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:02.938 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:24:03.261 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:14:55 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:03.326 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:24:03.345 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:03.345 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:03.346 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:03.367 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:24:03.688 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:14:55 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:03.753 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:24:03.772 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:03.772 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:03.773 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:03.794 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:24:04.117 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:14:56 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:04.183 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:24:04.202 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:04.202 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:04.202 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:04.225 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-osd 2026-02-25T08:24:04.547 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:14:56 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:04.612 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:24:04.631 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:04.632 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:04.632 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:04.653 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-osd-classic 2026-02-25T08:24:04.973 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:14:56 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:05.038 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:24:05.056 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:05.057 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:05.057 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:05.079 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-fuse 2026-02-25T08:24:05.399 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:14:57 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:05.483 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:05.484 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:05.484 INFO:teuthology.orchestra.run.trial169.stdout: Package Arch Version Repo Size 2026-02-25T08:24:05.484 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:05.484 INFO:teuthology.orchestra.run.trial169.stdout:Installing: 2026-02-25T08:24:05.484 INFO:teuthology.orchestra.run.trial169.stdout: ceph-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 914 k 2026-02-25T08:24:05.484 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:05.484 INFO:teuthology.orchestra.run.trial169.stdout:Transaction Summary 2026-02-25T08:24:05.485 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:05.485 INFO:teuthology.orchestra.run.trial169.stdout:Install 1 Package 2026-02-25T08:24:05.485 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:05.485 INFO:teuthology.orchestra.run.trial169.stdout:Total download size: 914 k 2026-02-25T08:24:05.485 INFO:teuthology.orchestra.run.trial169.stdout:Installed size: 2.9 M 2026-02-25T08:24:05.485 INFO:teuthology.orchestra.run.trial169.stdout:Downloading Packages: 2026-02-25T08:24:07.163 INFO:teuthology.orchestra.run.trial169.stdout:ceph-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 544 kB/s | 914 kB 00:01 2026-02-25T08:24:07.164 INFO:teuthology.orchestra.run.trial169.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:24:07.164 INFO:teuthology.orchestra.run.trial169.stdout:Total 544 kB/s | 914 kB 00:01 2026-02-25T08:24:07.165 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction check 2026-02-25T08:24:07.170 INFO:teuthology.orchestra.run.trial169.stdout:Transaction check succeeded. 2026-02-25T08:24:07.170 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction test 2026-02-25T08:24:07.203 INFO:teuthology.orchestra.run.trial169.stdout:Transaction test succeeded. 2026-02-25T08:24:07.203 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction 2026-02-25T08:24:07.295 INFO:teuthology.orchestra.run.trial169.stdout: Preparing : 1/1 2026-02-25T08:24:07.315 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:07.562 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:07.625 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:07.626 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:07.626 INFO:teuthology.orchestra.run.trial169.stdout:Installed: 2026-02-25T08:24:07.626 INFO:teuthology.orchestra.run.trial169.stdout: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:24:07.626 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:07.626 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:07.657 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-volume 2026-02-25T08:24:07.980 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:14:59 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:08.045 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:24:08.064 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:08.065 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:08.065 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:08.087 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install librados-devel 2026-02-25T08:24:08.404 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:15:00 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:08.488 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:08.489 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:08.489 INFO:teuthology.orchestra.run.trial169.stdout: Package Arch Version Repo Size 2026-02-25T08:24:08.490 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:08.490 INFO:teuthology.orchestra.run.trial169.stdout:Installing: 2026-02-25T08:24:08.490 INFO:teuthology.orchestra.run.trial169.stdout: librados-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 124 k 2026-02-25T08:24:08.490 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:08.490 INFO:teuthology.orchestra.run.trial169.stdout:Transaction Summary 2026-02-25T08:24:08.490 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:08.490 INFO:teuthology.orchestra.run.trial169.stdout:Install 1 Package 2026-02-25T08:24:08.490 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:08.490 INFO:teuthology.orchestra.run.trial169.stdout:Total download size: 124 k 2026-02-25T08:24:08.490 INFO:teuthology.orchestra.run.trial169.stdout:Installed size: 434 k 2026-02-25T08:24:08.490 INFO:teuthology.orchestra.run.trial169.stdout:Downloading Packages: 2026-02-25T08:24:08.984 INFO:teuthology.orchestra.run.trial169.stdout:librados-devel-20.3.0-5338.g6aeb6975.el9.x86_64 250 kB/s | 124 kB 00:00 2026-02-25T08:24:08.985 INFO:teuthology.orchestra.run.trial169.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:24:08.985 INFO:teuthology.orchestra.run.trial169.stdout:Total 249 kB/s | 124 kB 00:00 2026-02-25T08:24:08.985 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction check 2026-02-25T08:24:08.988 INFO:teuthology.orchestra.run.trial169.stdout:Transaction check succeeded. 2026-02-25T08:24:08.988 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction test 2026-02-25T08:24:09.019 INFO:teuthology.orchestra.run.trial169.stdout:Transaction test succeeded. 2026-02-25T08:24:09.020 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction 2026-02-25T08:24:09.074 INFO:teuthology.orchestra.run.trial169.stdout: Preparing : 1/1 2026-02-25T08:24:09.091 INFO:teuthology.orchestra.run.trial169.stdout: Installing : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:09.176 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:09.240 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:09.240 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:09.240 INFO:teuthology.orchestra.run.trial169.stdout:Installed: 2026-02-25T08:24:09.240 INFO:teuthology.orchestra.run.trial169.stdout: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:24:09.240 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:09.240 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:09.269 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install libcephfs2 2026-02-25T08:24:09.588 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:15:01 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:09.653 INFO:teuthology.orchestra.run.trial169.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:24:09.672 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:09.673 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:09.673 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:09.694 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install libcephfs-devel 2026-02-25T08:24:10.011 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:15:02 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:10.095 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:10.096 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:10.096 INFO:teuthology.orchestra.run.trial169.stdout: Package Arch Version Repo Size 2026-02-25T08:24:10.096 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:10.096 INFO:teuthology.orchestra.run.trial169.stdout:Installing: 2026-02-25T08:24:10.096 INFO:teuthology.orchestra.run.trial169.stdout: libcephfs-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:24:10.096 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:10.096 INFO:teuthology.orchestra.run.trial169.stdout:Transaction Summary 2026-02-25T08:24:10.097 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:10.097 INFO:teuthology.orchestra.run.trial169.stdout:Install 1 Package 2026-02-25T08:24:10.097 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:10.097 INFO:teuthology.orchestra.run.trial169.stdout:Total download size: 37 k 2026-02-25T08:24:10.097 INFO:teuthology.orchestra.run.trial169.stdout:Installed size: 171 k 2026-02-25T08:24:10.097 INFO:teuthology.orchestra.run.trial169.stdout:Downloading Packages: 2026-02-25T08:24:10.455 INFO:teuthology.orchestra.run.trial169.stdout:libcephfs-devel-20.3.0-5338.g6aeb6975.el9.x86_6 104 kB/s | 37 kB 00:00 2026-02-25T08:24:10.456 INFO:teuthology.orchestra.run.trial169.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:24:10.456 INFO:teuthology.orchestra.run.trial169.stdout:Total 103 kB/s | 37 kB 00:00 2026-02-25T08:24:10.456 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction check 2026-02-25T08:24:10.459 INFO:teuthology.orchestra.run.trial169.stdout:Transaction check succeeded. 2026-02-25T08:24:10.459 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction test 2026-02-25T08:24:10.461 INFO:teuthology.orchestra.run.trial169.stdout:Transaction test succeeded. 2026-02-25T08:24:10.462 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction 2026-02-25T08:24:10.484 INFO:teuthology.orchestra.run.trial169.stdout: Preparing : 1/1 2026-02-25T08:24:10.504 INFO:teuthology.orchestra.run.trial169.stdout: Installing : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:10.570 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:10.634 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:10.634 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:10.634 INFO:teuthology.orchestra.run.trial169.stdout:Installed: 2026-02-25T08:24:10.634 INFO:teuthology.orchestra.run.trial169.stdout: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:24:10.634 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:10.634 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:10.661 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install librados2 2026-02-25T08:24:10.980 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:15:02 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:11.049 INFO:teuthology.orchestra.run.trial169.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:24:11.066 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:11.067 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:11.067 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:11.089 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install librbd1 2026-02-25T08:24:11.407 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:15:03 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:11.473 INFO:teuthology.orchestra.run.trial169.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:24:11.490 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:11.491 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:11.491 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:11.513 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install python3-rados 2026-02-25T08:24:11.834 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:15:03 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:11.899 INFO:teuthology.orchestra.run.trial169.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:24:11.917 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:11.918 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:11.918 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:11.940 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install python3-rgw 2026-02-25T08:24:12.256 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:15:04 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:12.321 INFO:teuthology.orchestra.run.trial169.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:24:12.339 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:12.339 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:12.339 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:12.360 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install python3-cephfs 2026-02-25T08:24:12.681 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:15:04 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:12.747 INFO:teuthology.orchestra.run.trial169.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:24:12.764 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:12.765 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:12.765 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:12.786 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install python3-rbd 2026-02-25T08:24:13.107 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:15:05 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:13.173 INFO:teuthology.orchestra.run.trial169.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:24:13.190 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:13.191 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:13.191 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:13.212 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install rbd-fuse 2026-02-25T08:24:13.529 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:15:05 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:13.613 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:13.613 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:13.614 INFO:teuthology.orchestra.run.trial169.stdout: Package Architecture Version Repository Size 2026-02-25T08:24:13.614 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:13.614 INFO:teuthology.orchestra.run.trial169.stdout:Installing: 2026-02-25T08:24:13.614 INFO:teuthology.orchestra.run.trial169.stdout: rbd-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 86 k 2026-02-25T08:24:13.614 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:13.614 INFO:teuthology.orchestra.run.trial169.stdout:Transaction Summary 2026-02-25T08:24:13.614 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:13.614 INFO:teuthology.orchestra.run.trial169.stdout:Install 1 Package 2026-02-25T08:24:13.614 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:13.615 INFO:teuthology.orchestra.run.trial169.stdout:Total download size: 86 k 2026-02-25T08:24:13.615 INFO:teuthology.orchestra.run.trial169.stdout:Installed size: 236 k 2026-02-25T08:24:13.615 INFO:teuthology.orchestra.run.trial169.stdout:Downloading Packages: 2026-02-25T08:24:14.339 INFO:teuthology.orchestra.run.trial169.stdout:rbd-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 119 kB/s | 86 kB 00:00 2026-02-25T08:24:14.340 INFO:teuthology.orchestra.run.trial169.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:24:14.340 INFO:teuthology.orchestra.run.trial169.stdout:Total 119 kB/s | 86 kB 00:00 2026-02-25T08:24:14.340 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction check 2026-02-25T08:24:14.345 INFO:teuthology.orchestra.run.trial169.stdout:Transaction check succeeded. 2026-02-25T08:24:14.345 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction test 2026-02-25T08:24:14.376 INFO:teuthology.orchestra.run.trial169.stdout:Transaction test succeeded. 2026-02-25T08:24:14.376 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction 2026-02-25T08:24:14.427 INFO:teuthology.orchestra.run.trial169.stdout: Preparing : 1/1 2026-02-25T08:24:14.445 INFO:teuthology.orchestra.run.trial169.stdout: Installing : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:14.532 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:14.596 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:14.596 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:14.596 INFO:teuthology.orchestra.run.trial169.stdout:Installed: 2026-02-25T08:24:14.596 INFO:teuthology.orchestra.run.trial169.stdout: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:24:14.596 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:14.596 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:14.623 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install rbd-mirror 2026-02-25T08:24:14.942 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:15:06 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:15.026 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:15.026 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:15.027 INFO:teuthology.orchestra.run.trial169.stdout: Package Arch Version Repo Size 2026-02-25T08:24:15.027 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:15.027 INFO:teuthology.orchestra.run.trial169.stdout:Installing: 2026-02-25T08:24:15.027 INFO:teuthology.orchestra.run.trial169.stdout: rbd-mirror x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.6 M 2026-02-25T08:24:15.027 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:15.027 INFO:teuthology.orchestra.run.trial169.stdout:Transaction Summary 2026-02-25T08:24:15.027 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:15.027 INFO:teuthology.orchestra.run.trial169.stdout:Install 1 Package 2026-02-25T08:24:15.027 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:15.027 INFO:teuthology.orchestra.run.trial169.stdout:Total download size: 2.6 M 2026-02-25T08:24:15.027 INFO:teuthology.orchestra.run.trial169.stdout:Installed size: 11 M 2026-02-25T08:24:15.028 INFO:teuthology.orchestra.run.trial169.stdout:Downloading Packages: 2026-02-25T08:24:23.021 INFO:teuthology.orchestra.run.trial169.stdout:rbd-mirror-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 332 kB/s | 2.6 MB 00:07 2026-02-25T08:24:23.022 INFO:teuthology.orchestra.run.trial169.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:24:23.023 INFO:teuthology.orchestra.run.trial169.stdout:Total 332 kB/s | 2.6 MB 00:07 2026-02-25T08:24:23.023 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction check 2026-02-25T08:24:23.028 INFO:teuthology.orchestra.run.trial169.stdout:Transaction check succeeded. 2026-02-25T08:24:23.029 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction test 2026-02-25T08:24:23.067 INFO:teuthology.orchestra.run.trial169.stdout:Transaction test succeeded. 2026-02-25T08:24:23.067 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction 2026-02-25T08:24:23.235 INFO:teuthology.orchestra.run.trial169.stdout: Preparing : 1/1 2026-02-25T08:24:23.237 INFO:teuthology.orchestra.run.trial169.stdout: Installing : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:23.260 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:23.260 INFO:teuthology.orchestra.run.trial169.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:24:23.261 INFO:teuthology.orchestra.run.trial169.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-25T08:24:23.261 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:24:23.261 INFO:teuthology.orchestra.run.trial169.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:24:23.261 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:23.580 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:23.581 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:23.581 INFO:teuthology.orchestra.run.trial169.stdout:Installed: 2026-02-25T08:24:23.581 INFO:teuthology.orchestra.run.trial169.stdout: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:24:23.581 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:23.581 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:23.610 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install rbd-nbd 2026-02-25T08:24:23.939 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:15:15 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:24.022 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:24.023 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:24.023 INFO:teuthology.orchestra.run.trial169.stdout: Package Architecture Version Repository Size 2026-02-25T08:24:24.023 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:24.023 INFO:teuthology.orchestra.run.trial169.stdout:Installing: 2026-02-25T08:24:24.023 INFO:teuthology.orchestra.run.trial169.stdout: rbd-nbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 166 k 2026-02-25T08:24:24.023 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:24.024 INFO:teuthology.orchestra.run.trial169.stdout:Transaction Summary 2026-02-25T08:24:24.024 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:24.024 INFO:teuthology.orchestra.run.trial169.stdout:Install 1 Package 2026-02-25T08:24:24.024 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:24.024 INFO:teuthology.orchestra.run.trial169.stdout:Total download size: 166 k 2026-02-25T08:24:24.024 INFO:teuthology.orchestra.run.trial169.stdout:Installed size: 496 k 2026-02-25T08:24:24.024 INFO:teuthology.orchestra.run.trial169.stdout:Downloading Packages: 2026-02-25T08:24:24.656 INFO:teuthology.orchestra.run.trial169.stdout:rbd-nbd-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 263 kB/s | 166 kB 00:00 2026-02-25T08:24:24.656 INFO:teuthology.orchestra.run.trial169.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:24:24.656 INFO:teuthology.orchestra.run.trial169.stdout:Total 263 kB/s | 166 kB 00:00 2026-02-25T08:24:24.657 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction check 2026-02-25T08:24:24.661 INFO:teuthology.orchestra.run.trial169.stdout:Transaction check succeeded. 2026-02-25T08:24:24.662 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction test 2026-02-25T08:24:24.693 INFO:teuthology.orchestra.run.trial169.stdout:Transaction test succeeded. 2026-02-25T08:24:24.693 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction 2026-02-25T08:24:24.750 INFO:teuthology.orchestra.run.trial169.stdout: Preparing : 1/1 2026-02-25T08:24:24.767 INFO:teuthology.orchestra.run.trial169.stdout: Installing : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:24.855 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:24.919 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:24:24.919 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:24.919 INFO:teuthology.orchestra.run.trial169.stdout:Installed: 2026-02-25T08:24:24.919 INFO:teuthology.orchestra.run.trial169.stdout: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:24:24.919 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:24.919 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:24.948 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-osd-crimson 2026-02-25T08:24:25.265 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:15:17 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:25.330 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:24:25.347 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:25.348 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:24:25.348 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:24:25.370 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-25T08:24:25.690 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:15:17 ago on Wed 25 Feb 2026 08:09:08 AM UTC. 2026-02-25T08:24:25.773 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:24:25.774 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:25.774 INFO:teuthology.orchestra.run.trial169.stdout: Package Arch Version Repo Size 2026-02-25T08:24:25.774 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:25.774 INFO:teuthology.orchestra.run.trial169.stdout:Installing: 2026-02-25T08:24:25.774 INFO:teuthology.orchestra.run.trial169.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 433 M 2026-02-25T08:24:25.774 INFO:teuthology.orchestra.run.trial169.stdout:Installing weak dependencies: 2026-02-25T08:24:25.774 INFO:teuthology.orchestra.run.trial169.stdout: ceph-debugsource x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 12 M 2026-02-25T08:24:25.775 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:25.775 INFO:teuthology.orchestra.run.trial169.stdout:Transaction Summary 2026-02-25T08:24:25.775 INFO:teuthology.orchestra.run.trial169.stdout:================================================================================ 2026-02-25T08:24:25.775 INFO:teuthology.orchestra.run.trial169.stdout:Install 2 Packages 2026-02-25T08:24:25.775 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:24:25.775 INFO:teuthology.orchestra.run.trial169.stdout:Total download size: 445 M 2026-02-25T08:24:25.775 INFO:teuthology.orchestra.run.trial169.stdout:Installed size: 2.7 G 2026-02-25T08:24:25.775 INFO:teuthology.orchestra.run.trial169.stdout:Downloading Packages: 2026-02-25T08:25:02.503 INFO:teuthology.orchestra.run.trial169.stdout:(1/2): ceph-debugsource-20.3.0-5338.g6aeb6975.e 336 kB/s | 12 MB 00:36 2026-02-25T08:25:18.438 INFO:teuthology.orchestra.run.trial183.stdout:(95/96): ceph-mgr-dashboard-20.3.0-5338.g6aeb69 301 kB/s | 51 MB 02:52 2026-02-25T08:27:15.646 INFO:teuthology.orchestra.run.trial183.stdout:(96/96): ceph-osd-crimson-20.3.0-5338.g6aeb6975 296 kB/s | 88 MB 05:02 2026-02-25T08:27:15.648 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:27:15.648 INFO:teuthology.orchestra.run.trial183.stdout:Total 745 kB/s | 226 MB 05:10 2026-02-25T08:27:16.026 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-25T08:27:16.063 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-25T08:27:16.063 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-25T08:27:16.866 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-25T08:27:16.866 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-25T08:27:17.700 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-25T08:27:17.702 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-25T08:27:17.706 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-25T08:27:17.727 INFO:teuthology.orchestra.run.trial183.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-25T08:27:17.760 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pyasn1-0.4.8-7.el9.noarch 4/96 2026-02-25T08:27:17.767 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-25T08:27:17.769 INFO:teuthology.orchestra.run.trial183.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-25T08:27:17.774 INFO:teuthology.orchestra.run.trial183.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-25T08:27:17.797 INFO:teuthology.orchestra.run.trial183.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-25T08:27:17.809 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-25T08:27:17.812 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-25T08:27:17.820 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-25T08:27:17.822 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-25T08:27:17.833 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-25T08:27:17.839 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-25T08:27:17.864 INFO:teuthology.orchestra.run.trial183.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-25T08:27:17.867 INFO:teuthology.orchestra.run.trial183.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-25T08:27:17.917 INFO:teuthology.orchestra.run.trial183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-25T08:27:17.930 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pyasn1-modules-0.4.8-7.el9.noarch 18/96 2026-02-25T08:27:17.932 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-25T08:27:17.937 INFO:teuthology.orchestra.run.trial183.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-25T08:27:17.942 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-25T08:27:17.947 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-25T08:27:17.950 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-25T08:27:17.964 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-25T08:27:17.969 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-25T08:27:17.974 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-25T08:27:17.985 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-25T08:27:17.994 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-25T08:27:18.000 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-25T08:27:18.020 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-25T08:27:18.031 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-25T08:27:18.046 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-25T08:27:18.051 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-25T08:27:18.059 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-25T08:27:18.099 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-25T08:27:18.423 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-25T08:27:18.433 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-25T08:27:18.437 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-25T08:27:18.444 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-25T08:27:18.445 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-25T08:27:18.473 INFO:teuthology.orchestra.run.trial183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-25T08:27:18.518 INFO:teuthology.orchestra.run.trial183.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-25T08:27:18.526 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-25T08:27:18.532 INFO:teuthology.orchestra.run.trial183.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-25T08:27:18.535 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-25T08:27:18.541 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-25T08:27:18.545 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-25T08:27:18.552 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-25T08:27:18.555 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-25T08:27:18.583 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-25T08:27:18.594 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-25T08:27:18.600 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-25T08:27:18.607 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-25T08:27:18.641 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-25T08:27:18.867 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-25T08:27:18.891 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-25T08:27:18.895 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-25T08:27:18.953 INFO:teuthology.orchestra.run.trial183.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-25T08:27:18.955 INFO:teuthology.orchestra.run.trial183.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-25T08:27:18.977 INFO:teuthology.orchestra.run.trial183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-25T08:27:19.305 INFO:teuthology.orchestra.run.trial183.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-25T08:27:19.379 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-25T08:27:20.055 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-25T08:27:20.072 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-25T08:27:20.082 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-25T08:27:20.088 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-25T08:27:20.102 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-25T08:27:20.118 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-25T08:27:20.194 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-25T08:27:20.203 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-25T08:27:20.226 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-25T08:27:20.256 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-25T08:27:20.307 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-25T08:27:20.313 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-25T08:27:20.316 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-25T08:27:20.317 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-25T08:27:20.336 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:27:20.352 INFO:teuthology.orchestra.run.trial183.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-25T08:27:20.352 INFO:teuthology.orchestra.run.trial183.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-25T08:27:20.353 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:20.353 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:27:20.369 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-25T08:27:20.369 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-25T08:27:20.370 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:20.404 INFO:teuthology.orchestra.run.trial183.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-25T08:27:21.015 INFO:teuthology.orchestra.run.trial183.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-25T08:27:26.516 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 80/96 2026-02-25T08:27:26.579 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 81/96 2026-02-25T08:27:26.581 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:27:26.595 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 82/96 2026-02-25T08:27:26.595 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:27:26.595 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-25T08:27:26.595 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:26.606 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:27:26.630 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 83/96 2026-02-25T08:27:26.630 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:27:26.630 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-25T08:27:26.630 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:27:26.630 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-25T08:27:26.630 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:26.687 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:27:26.689 INFO:teuthology.orchestra.run.trial183.stdout: Installing : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 84/96 2026-02-25T08:27:26.698 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 85/96 2026-02-25T08:27:26.722 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 86/96 2026-02-25T08:27:26.724 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:27:29.746 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 87/96 2026-02-25T08:27:29.752 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:27:30.223 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 88/96 2026-02-25T08:27:30.224 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:27:30.236 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 89/96 2026-02-25T08:27:30.237 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:27:30.298 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 90/96 2026-02-25T08:27:30.340 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 91/96 2026-02-25T08:27:30.341 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:27:30.362 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 92/96 2026-02-25T08:27:30.362 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:27:30.363 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-25T08:27:30.363 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:27:30.363 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-25T08:27:30.363 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:30.374 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:27:30.594 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 93/96 2026-02-25T08:27:30.595 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:27:30.616 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 94/96 2026-02-25T08:27:30.616 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:27:30.616 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-25T08:27:30.616 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:27:30.616 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-25T08:27:30.616 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:30.701 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:27:30.721 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 95/96 2026-02-25T08:27:30.721 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:27:30.721 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-25T08:27:30.721 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:27:30.721 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-25T08:27:30.721 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:30.722 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:27:30.728 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:27:30.753 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 96/96 2026-02-25T08:27:32.140 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 96/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 3/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 4/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 5/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_6 6/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_6 7/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el 8/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarc 9/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noa 10/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6 11/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noa 12/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9. 13/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 14/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9 15/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 16/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 17/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-25T08:27:32.141 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-25T08:27:32.147 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-25T08:27:32.147 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-25T08:27:32.147 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-25T08:27:32.147 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-25T08:27:32.147 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-25T08:27:32.147 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-25T08:27:32.148 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-25T08:27:32.148 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-25T08:27:32.148 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-25T08:27:32.148 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-25T08:27:32.148 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-25T08:27:32.148 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-25T08:27:32.148 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-25T08:27:32.148 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-25T08:27:32.148 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-25T08:27:32.148 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-25T08:27:32.148 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-25T08:27:32.148 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-25T08:27:32.148 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-25T08:27:32.149 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-25T08:27:32.149 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-25T08:27:32.149 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-25T08:27:32.149 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-25T08:27:32.149 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pyasn1-0.4.8-7.el9.noarch 50/96 2026-02-25T08:27:32.149 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pyasn1-modules-0.4.8-7.el9.noarch 51/96 2026-02-25T08:27:32.149 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-25T08:27:32.149 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-25T08:27:32.149 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-25T08:27:32.149 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-25T08:27:32.149 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-25T08:27:32.149 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-25T08:27:32.150 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-25T08:27:32.150 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-25T08:27:32.150 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-25T08:27:32.150 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-25T08:27:32.150 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-25T08:27:32.150 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-25T08:27:32.150 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-25T08:27:32.150 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-25T08:27:32.150 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-25T08:27:32.150 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-25T08:27:32.150 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-25T08:27:32.150 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-25T08:27:32.150 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-25T08:27:32.151 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-25T08:27:32.151 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-25T08:27:32.151 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-25T08:27:32.151 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-25T08:27:32.151 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-25T08:27:32.151 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-25T08:27:32.151 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-25T08:27:32.151 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-25T08:27:32.151 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-25T08:27:32.151 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-25T08:27:32.151 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-25T08:27:32.151 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-25T08:27:32.151 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-25T08:27:32.151 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-25T08:27:32.152 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-25T08:27:32.152 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-25T08:27:32.152 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-25T08:27:32.152 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-25T08:27:32.152 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-25T08:27:32.152 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-25T08:27:32.152 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-25T08:27:32.152 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-25T08:27:32.152 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-25T08:27:32.152 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-25T08:27:32.152 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-25T08:27:32.217 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-25T08:27:32.217 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:32.218 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-25T08:27:32.218 INFO:teuthology.orchestra.run.trial183.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-25T08:27:32.218 INFO:teuthology.orchestra.run.trial183.stdout: ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:27:32.218 INFO:teuthology.orchestra.run.trial183.stdout: ceph-grafana-dashboards-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:27:32.218 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mds-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:27:32.218 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:27:32.218 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:27:32.218 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:27:32.218 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:27:32.218 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-k8sevents-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:27:32.218 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-modules-core-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:27:32.218 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:27:32.219 INFO:teuthology.orchestra.run.trial183.stdout: ceph-mon-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:27:32.219 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:27:32.219 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:27:32.219 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:27:32.219 INFO:teuthology.orchestra.run.trial183.stdout: ceph-prometheus-alerts-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:27:32.219 INFO:teuthology.orchestra.run.trial183.stdout: ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:27:32.219 INFO:teuthology.orchestra.run.trial183.stdout: cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch 2026-02-25T08:27:32.219 INFO:teuthology.orchestra.run.trial183.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-25T08:27:32.219 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-25T08:27:32.219 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-25T08:27:32.219 INFO:teuthology.orchestra.run.trial183.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-25T08:27:32.219 INFO:teuthology.orchestra.run.trial183.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-25T08:27:32.220 INFO:teuthology.orchestra.run.trial183.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-25T08:27:32.220 INFO:teuthology.orchestra.run.trial183.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-25T08:27:32.220 INFO:teuthology.orchestra.run.trial183.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-25T08:27:32.220 INFO:teuthology.orchestra.run.trial183.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-25T08:27:32.220 INFO:teuthology.orchestra.run.trial183.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:27:32.220 INFO:teuthology.orchestra.run.trial183.stdout: lua-5.4.4-4.el9.x86_64 2026-02-25T08:27:32.220 INFO:teuthology.orchestra.run.trial183.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-25T08:27:32.220 INFO:teuthology.orchestra.run.trial183.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-25T08:27:32.220 INFO:teuthology.orchestra.run.trial183.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-25T08:27:32.220 INFO:teuthology.orchestra.run.trial183.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-25T08:27:32.220 INFO:teuthology.orchestra.run.trial183.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-25T08:27:32.220 INFO:teuthology.orchestra.run.trial183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-25T08:27:32.220 INFO:teuthology.orchestra.run.trial183.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-25T08:27:32.221 INFO:teuthology.orchestra.run.trial183.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-25T08:27:32.221 INFO:teuthology.orchestra.run.trial183.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-25T08:27:32.221 INFO:teuthology.orchestra.run.trial183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-25T08:27:32.221 INFO:teuthology.orchestra.run.trial183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-25T08:27:32.221 INFO:teuthology.orchestra.run.trial183.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-25T08:27:32.221 INFO:teuthology.orchestra.run.trial183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-25T08:27:32.221 INFO:teuthology.orchestra.run.trial183.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-25T08:27:32.221 INFO:teuthology.orchestra.run.trial183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-25T08:27:32.221 INFO:teuthology.orchestra.run.trial183.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-25T08:27:32.221 INFO:teuthology.orchestra.run.trial183.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-25T08:27:32.221 INFO:teuthology.orchestra.run.trial183.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-25T08:27:32.221 INFO:teuthology.orchestra.run.trial183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-25T08:27:32.221 INFO:teuthology.orchestra.run.trial183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-25T08:27:32.222 INFO:teuthology.orchestra.run.trial183.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-25T08:27:32.222 INFO:teuthology.orchestra.run.trial183.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-25T08:27:32.222 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-25T08:27:32.222 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-25T08:27:32.222 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-25T08:27:32.222 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-25T08:27:32.222 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-25T08:27:32.222 INFO:teuthology.orchestra.run.trial183.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-25T08:27:32.222 INFO:teuthology.orchestra.run.trial183.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-25T08:27:32.222 INFO:teuthology.orchestra.run.trial183.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-25T08:27:32.222 INFO:teuthology.orchestra.run.trial183.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-25T08:27:32.222 INFO:teuthology.orchestra.run.trial183.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-25T08:27:32.222 INFO:teuthology.orchestra.run.trial183.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-25T08:27:32.223 INFO:teuthology.orchestra.run.trial183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-25T08:27:32.223 INFO:teuthology.orchestra.run.trial183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-25T08:27:32.223 INFO:teuthology.orchestra.run.trial183.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-25T08:27:32.223 INFO:teuthology.orchestra.run.trial183.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-25T08:27:32.223 INFO:teuthology.orchestra.run.trial183.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-25T08:27:32.223 INFO:teuthology.orchestra.run.trial183.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-25T08:27:32.223 INFO:teuthology.orchestra.run.trial183.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-25T08:27:32.223 INFO:teuthology.orchestra.run.trial183.stdout: python3-ply-3.11-14.el9.noarch 2026-02-25T08:27:32.223 INFO:teuthology.orchestra.run.trial183.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-25T08:27:32.223 INFO:teuthology.orchestra.run.trial183.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-25T08:27:32.223 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-25T08:27:32.223 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyasn1-0.4.8-7.el9.noarch 2026-02-25T08:27:32.223 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyasn1-modules-0.4.8-7.el9.noarch 2026-02-25T08:27:32.224 INFO:teuthology.orchestra.run.trial183.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-25T08:27:32.224 INFO:teuthology.orchestra.run.trial183.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-25T08:27:32.224 INFO:teuthology.orchestra.run.trial183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-25T08:27:32.224 INFO:teuthology.orchestra.run.trial183.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-25T08:27:32.224 INFO:teuthology.orchestra.run.trial183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-25T08:27:32.224 INFO:teuthology.orchestra.run.trial183.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-25T08:27:32.224 INFO:teuthology.orchestra.run.trial183.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-25T08:27:32.224 INFO:teuthology.orchestra.run.trial183.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-25T08:27:32.224 INFO:teuthology.orchestra.run.trial183.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-25T08:27:32.224 INFO:teuthology.orchestra.run.trial183.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-25T08:27:32.224 INFO:teuthology.orchestra.run.trial183.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-25T08:27:32.224 INFO:teuthology.orchestra.run.trial183.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-25T08:27:32.224 INFO:teuthology.orchestra.run.trial183.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-25T08:27:32.225 INFO:teuthology.orchestra.run.trial183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-25T08:27:32.225 INFO:teuthology.orchestra.run.trial183.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-25T08:27:32.225 INFO:teuthology.orchestra.run.trial183.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-25T08:27:32.225 INFO:teuthology.orchestra.run.trial183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-25T08:27:32.225 INFO:teuthology.orchestra.run.trial183.stdout: unzip-6.0-59.el9.x86_64 2026-02-25T08:27:32.225 INFO:teuthology.orchestra.run.trial183.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-25T08:27:32.225 INFO:teuthology.orchestra.run.trial183.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-25T08:27:32.225 INFO:teuthology.orchestra.run.trial183.stdout: zip-3.0-35.el9.x86_64 2026-02-25T08:27:32.225 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:32.225 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:32.282 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-base 2026-02-25T08:27:32.608 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:09 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:32.673 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:27:32.692 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:32.693 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:32.693 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:32.714 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install cephadm 2026-02-25T08:27:33.030 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:10 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:33.096 INFO:teuthology.orchestra.run.trial183.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:27:33.114 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:33.115 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:33.115 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:33.135 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:27:33.452 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:10 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:33.537 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:33.537 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:33.537 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-25T08:27:33.537 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:33.537 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-25T08:27:33.537 INFO:teuthology.orchestra.run.trial183.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 149 k 2026-02-25T08:27:33.537 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:33.537 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-25T08:27:33.538 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:33.538 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-25T08:27:33.538 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:33.538 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 149 k 2026-02-25T08:27:33.538 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 455 k 2026-02-25T08:27:33.538 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-25T08:27:34.168 INFO:teuthology.orchestra.run.trial183.stdout:ceph-immutable-object-cache-20.3.0-5338.g6aeb69 237 kB/s | 149 kB 00:00 2026-02-25T08:27:34.169 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:27:34.169 INFO:teuthology.orchestra.run.trial183.stdout:Total 237 kB/s | 149 kB 00:00 2026-02-25T08:27:34.169 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-25T08:27:34.174 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-25T08:27:34.174 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-25T08:27:34.205 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-25T08:27:34.205 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-25T08:27:34.261 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-25T08:27:34.263 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:27:34.286 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:27:34.286 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:27:34.286 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-25T08:27:34.286 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:34.581 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975. 1/1 2026-02-25T08:27:34.581 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:34.581 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-25T08:27:34.581 INFO:teuthology.orchestra.run.trial183.stdout: ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:27:34.581 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:34.581 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:34.607 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr 2026-02-25T08:27:34.930 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:11 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:34.997 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:27:35.016 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:35.017 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:35.017 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:35.038 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:27:35.352 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:12 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:35.416 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:27:35.435 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:35.436 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:35.436 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:35.456 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:27:35.774 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:12 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:35.840 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:27:35.858 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:35.859 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:35.859 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:35.881 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:27:36.200 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:13 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:36.265 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:27:36.284 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:36.285 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:36.285 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:36.306 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:27:36.622 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:13 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:36.687 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:27:36.705 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:36.706 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:36.706 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:36.727 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-osd 2026-02-25T08:27:37.045 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:14 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:37.111 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:27:37.130 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:37.131 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:37.131 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:37.152 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-osd-classic 2026-02-25T08:27:37.467 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:14 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:37.533 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:27:37.552 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:37.553 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:37.553 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:37.574 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-fuse 2026-02-25T08:27:37.897 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:14 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:37.982 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:37.982 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:37.983 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-25T08:27:37.983 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:37.983 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-25T08:27:37.983 INFO:teuthology.orchestra.run.trial183.stdout: ceph-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 914 k 2026-02-25T08:27:37.983 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:37.983 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-25T08:27:37.983 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:37.983 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-25T08:27:37.983 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:37.983 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 914 k 2026-02-25T08:27:37.984 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 2.9 M 2026-02-25T08:27:37.984 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-25T08:27:42.152 INFO:teuthology.orchestra.run.trial183.stdout:ceph-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 219 kB/s | 914 kB 00:04 2026-02-25T08:27:42.152 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:27:42.153 INFO:teuthology.orchestra.run.trial183.stdout:Total 219 kB/s | 914 kB 00:04 2026-02-25T08:27:42.153 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-25T08:27:42.158 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-25T08:27:42.158 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-25T08:27:42.191 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-25T08:27:42.191 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-25T08:27:42.283 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-25T08:27:42.300 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:42.551 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:42.613 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:42.613 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:42.614 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-25T08:27:42.614 INFO:teuthology.orchestra.run.trial183.stdout: ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:27:42.614 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:42.614 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:42.642 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-volume 2026-02-25T08:27:42.962 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:19 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:43.027 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:27:43.046 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:43.046 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:43.046 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:43.068 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librados-devel 2026-02-25T08:27:43.387 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:20 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:43.473 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:43.473 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:43.473 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-25T08:27:43.474 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:43.474 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-25T08:27:43.474 INFO:teuthology.orchestra.run.trial183.stdout: librados-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 124 k 2026-02-25T08:27:43.474 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:43.474 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-25T08:27:43.474 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:43.474 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-25T08:27:43.474 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:43.474 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 124 k 2026-02-25T08:27:43.474 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 434 k 2026-02-25T08:27:43.474 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-25T08:27:44.040 INFO:teuthology.orchestra.run.trial183.stdout:librados-devel-20.3.0-5338.g6aeb6975.el9.x86_64 218 kB/s | 124 kB 00:00 2026-02-25T08:27:44.040 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:27:44.040 INFO:teuthology.orchestra.run.trial183.stdout:Total 218 kB/s | 124 kB 00:00 2026-02-25T08:27:44.040 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-25T08:27:44.043 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-25T08:27:44.044 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-25T08:27:44.075 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-25T08:27:44.075 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-25T08:27:44.130 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-25T08:27:44.148 INFO:teuthology.orchestra.run.trial183.stdout: Installing : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:44.234 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:44.299 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:44.299 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:44.300 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-25T08:27:44.300 INFO:teuthology.orchestra.run.trial183.stdout: librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:27:44.300 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:44.300 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:44.328 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install libcephfs2 2026-02-25T08:27:44.645 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:21 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:44.711 INFO:teuthology.orchestra.run.trial183.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:27:44.730 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:44.730 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:44.730 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:44.751 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install libcephfs-devel 2026-02-25T08:27:45.072 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:22 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:45.157 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:45.157 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:45.157 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-25T08:27:45.157 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:45.157 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-25T08:27:45.158 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-devel x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 37 k 2026-02-25T08:27:45.158 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:45.158 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-25T08:27:45.158 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:45.158 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-25T08:27:45.158 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:45.158 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 37 k 2026-02-25T08:27:45.158 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 171 k 2026-02-25T08:27:45.158 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-25T08:27:45.516 INFO:teuthology.orchestra.run.trial183.stdout:libcephfs-devel-20.3.0-5338.g6aeb6975.el9.x86_6 104 kB/s | 37 kB 00:00 2026-02-25T08:27:45.517 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:27:45.517 INFO:teuthology.orchestra.run.trial183.stdout:Total 104 kB/s | 37 kB 00:00 2026-02-25T08:27:45.517 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-25T08:27:45.520 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-25T08:27:45.520 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-25T08:27:45.522 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-25T08:27:45.522 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-25T08:27:45.545 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-25T08:27:45.562 INFO:teuthology.orchestra.run.trial183.stdout: Installing : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:45.627 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:45.689 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:45.690 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:45.690 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-25T08:27:45.690 INFO:teuthology.orchestra.run.trial183.stdout: libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:27:45.690 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:45.690 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:45.717 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librados2 2026-02-25T08:27:46.037 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:23 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:46.102 INFO:teuthology.orchestra.run.trial183.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:27:46.120 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:46.120 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:46.120 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:46.142 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librbd1 2026-02-25T08:27:46.461 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:23 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:46.527 INFO:teuthology.orchestra.run.trial183.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:27:46.544 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:46.545 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:46.545 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:46.566 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rados 2026-02-25T08:27:46.883 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:23 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:46.950 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:27:46.967 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:46.968 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:46.968 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:46.990 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rgw 2026-02-25T08:27:47.305 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:24 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:47.370 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:27:47.388 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:47.388 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:47.388 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:47.409 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-cephfs 2026-02-25T08:27:47.726 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:24 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:47.793 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:27:47.810 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:47.811 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:47.811 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:47.832 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rbd 2026-02-25T08:27:48.148 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:25 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:48.214 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:27:48.233 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:48.234 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:48.234 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:48.255 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-fuse 2026-02-25T08:27:48.570 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:25 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:48.652 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:48.653 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:48.653 INFO:teuthology.orchestra.run.trial183.stdout: Package Architecture Version Repository Size 2026-02-25T08:27:48.653 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:48.653 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-25T08:27:48.653 INFO:teuthology.orchestra.run.trial183.stdout: rbd-fuse x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 86 k 2026-02-25T08:27:48.653 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:48.653 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-25T08:27:48.653 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:48.653 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-25T08:27:48.653 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:48.653 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 86 k 2026-02-25T08:27:48.653 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 236 k 2026-02-25T08:27:48.653 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-25T08:27:49.077 INFO:teuthology.orchestra.run.trial183.stdout:rbd-fuse-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 204 kB/s | 86 kB 00:00 2026-02-25T08:27:49.078 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:27:49.078 INFO:teuthology.orchestra.run.trial183.stdout:Total 203 kB/s | 86 kB 00:00 2026-02-25T08:27:49.078 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-25T08:27:49.082 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-25T08:27:49.082 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-25T08:27:49.113 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-25T08:27:49.114 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-25T08:27:49.165 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-25T08:27:49.183 INFO:teuthology.orchestra.run.trial183.stdout: Installing : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:49.271 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:49.335 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:49.335 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:49.335 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-25T08:27:49.335 INFO:teuthology.orchestra.run.trial183.stdout: rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:27:49.335 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:49.335 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:49.364 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-mirror 2026-02-25T08:27:49.686 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:26 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:49.771 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:49.771 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:49.771 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-25T08:27:49.771 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:49.771 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-25T08:27:49.771 INFO:teuthology.orchestra.run.trial183.stdout: rbd-mirror x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 2.6 M 2026-02-25T08:27:49.771 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:49.771 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-25T08:27:49.771 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:49.771 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-25T08:27:49.772 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:49.772 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 2.6 M 2026-02-25T08:27:49.772 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 11 M 2026-02-25T08:27:49.772 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-25T08:27:55.891 INFO:teuthology.orchestra.run.trial183.stdout:rbd-mirror-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 434 kB/s | 2.6 MB 00:06 2026-02-25T08:27:55.891 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:27:55.892 INFO:teuthology.orchestra.run.trial183.stdout:Total 434 kB/s | 2.6 MB 00:06 2026-02-25T08:27:55.893 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-25T08:27:55.898 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-25T08:27:55.898 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-25T08:27:55.937 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-25T08:27:55.937 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-25T08:27:56.110 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-25T08:27:56.111 INFO:teuthology.orchestra.run.trial183.stdout: Installing : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:56.135 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:56.135 INFO:teuthology.orchestra.run.trial183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-25T08:27:56.135 INFO:teuthology.orchestra.run.trial183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-25T08:27:56.135 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:27:56.135 INFO:teuthology.orchestra.run.trial183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-25T08:27:56.135 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:56.455 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:56.455 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:56.456 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-25T08:27:56.456 INFO:teuthology.orchestra.run.trial183.stdout: rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:27:56.456 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:56.456 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:56.483 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-nbd 2026-02-25T08:27:56.811 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:33 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:56.895 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:56.896 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:56.896 INFO:teuthology.orchestra.run.trial183.stdout: Package Architecture Version Repository Size 2026-02-25T08:27:56.896 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:56.896 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-25T08:27:56.897 INFO:teuthology.orchestra.run.trial183.stdout: rbd-nbd x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 166 k 2026-02-25T08:27:56.897 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:56.897 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-25T08:27:56.897 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:56.897 INFO:teuthology.orchestra.run.trial183.stdout:Install 1 Package 2026-02-25T08:27:56.897 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:56.897 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 166 k 2026-02-25T08:27:56.897 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 496 k 2026-02-25T08:27:56.897 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-25T08:27:57.488 INFO:teuthology.orchestra.run.trial183.stdout:rbd-nbd-20.3.0-5338.g6aeb6975.el9.x86_64.rpm 282 kB/s | 166 kB 00:00 2026-02-25T08:27:57.488 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:27:57.488 INFO:teuthology.orchestra.run.trial183.stdout:Total 281 kB/s | 166 kB 00:00 2026-02-25T08:27:57.488 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-25T08:27:57.493 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-25T08:27:57.493 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-25T08:27:57.524 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-25T08:27:57.525 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-25T08:27:57.581 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-25T08:27:57.599 INFO:teuthology.orchestra.run.trial183.stdout: Installing : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:57.690 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:57.753 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/1 2026-02-25T08:27:57.753 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:57.753 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-25T08:27:57.754 INFO:teuthology.orchestra.run.trial183.stdout: rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:27:57.754 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:57.754 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:57.782 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-osd-crimson 2026-02-25T08:27:58.099 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:35 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:58.164 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-osd-crimson-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:27:58.182 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:58.182 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:27:58.183 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:27:58.204 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-osd-crimson-debuginfo 2026-02-25T08:27:58.531 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:18:35 ago on Wed 25 Feb 2026 08:09:23 AM UTC. 2026-02-25T08:27:58.615 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:27:58.616 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:58.616 INFO:teuthology.orchestra.run.trial183.stdout: Package Arch Version Repo Size 2026-02-25T08:27:58.616 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:58.616 INFO:teuthology.orchestra.run.trial183.stdout:Installing: 2026-02-25T08:27:58.617 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-crimson-debuginfo x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 433 M 2026-02-25T08:27:58.617 INFO:teuthology.orchestra.run.trial183.stdout:Installing weak dependencies: 2026-02-25T08:27:58.617 INFO:teuthology.orchestra.run.trial183.stdout: ceph-debugsource x86_64 2:20.3.0-5338.g6aeb6975.el9 ceph 12 M 2026-02-25T08:27:58.617 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:58.617 INFO:teuthology.orchestra.run.trial183.stdout:Transaction Summary 2026-02-25T08:27:58.617 INFO:teuthology.orchestra.run.trial183.stdout:================================================================================ 2026-02-25T08:27:58.617 INFO:teuthology.orchestra.run.trial183.stdout:Install 2 Packages 2026-02-25T08:27:58.617 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:27:58.617 INFO:teuthology.orchestra.run.trial183.stdout:Total download size: 445 M 2026-02-25T08:27:58.617 INFO:teuthology.orchestra.run.trial183.stdout:Installed size: 2.7 G 2026-02-25T08:27:58.617 INFO:teuthology.orchestra.run.trial183.stdout:Downloading Packages: 2026-02-25T08:28:34.597 INFO:teuthology.orchestra.run.trial183.stdout:(1/2): ceph-debugsource-20.3.0-5338.g6aeb6975.e 343 kB/s | 12 MB 00:35 2026-02-25T08:38:45.247 INFO:teuthology.orchestra.run.trial183.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5338.g 685 kB/s | 433 MB 10:46 2026-02-25T08:38:45.248 INFO:teuthology.orchestra.run.trial183.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:38:45.249 INFO:teuthology.orchestra.run.trial183.stdout:Total 704 kB/s | 445 MB 10:46 2026-02-25T08:38:45.249 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction check 2026-02-25T08:38:45.255 INFO:teuthology.orchestra.run.trial183.stdout:Transaction check succeeded. 2026-02-25T08:38:45.255 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction test 2026-02-25T08:38:46.631 INFO:teuthology.orchestra.run.trial183.stdout:Transaction test succeeded. 2026-02-25T08:38:46.631 INFO:teuthology.orchestra.run.trial183.stdout:Running transaction 2026-02-25T08:38:50.225 INFO:teuthology.orchestra.run.trial183.stdout: Preparing : 1/1 2026-02-25T08:39:12.892 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:39:12.912 INFO:teuthology.orchestra.run.trial183.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:39:14.800 INFO:teuthology.orchestra.run.trial183.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:39:14.801 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:39:14.865 INFO:teuthology.orchestra.run.trial183.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:39:14.865 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:39:14.866 INFO:teuthology.orchestra.run.trial183.stdout:Installed: 2026-02-25T08:39:14.866 INFO:teuthology.orchestra.run.trial183.stdout: ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:39:14.866 INFO:teuthology.orchestra.run.trial183.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:39:14.866 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:39:14.866 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:39:14.947 DEBUG:teuthology.parallel:result is None 2026-02-25T08:39:48.978 INFO:teuthology.orchestra.run.trial169.stdout:(2/2): ceph-osd-crimson-debuginfo-20.3.0-5338.g 480 kB/s | 433 MB 15:23 2026-02-25T08:39:48.979 INFO:teuthology.orchestra.run.trial169.stdout:-------------------------------------------------------------------------------- 2026-02-25T08:39:48.979 INFO:teuthology.orchestra.run.trial169.stdout:Total 493 kB/s | 445 MB 15:23 2026-02-25T08:39:48.979 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction check 2026-02-25T08:39:48.985 INFO:teuthology.orchestra.run.trial169.stdout:Transaction check succeeded. 2026-02-25T08:39:48.985 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction test 2026-02-25T08:39:50.366 INFO:teuthology.orchestra.run.trial169.stdout:Transaction test succeeded. 2026-02-25T08:39:50.366 INFO:teuthology.orchestra.run.trial169.stdout:Running transaction 2026-02-25T08:39:53.977 INFO:teuthology.orchestra.run.trial169.stdout: Preparing : 1/1 2026-02-25T08:40:16.232 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:40:16.318 INFO:teuthology.orchestra.run.trial169.stdout: Installing : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:40:16.469 INFO:teuthology.orchestra.run.trial169.stdout: Running scriptlet: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:40:16.470 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 1/2 2026-02-25T08:40:16.534 INFO:teuthology.orchestra.run.trial169.stdout: Verifying : ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.e 2/2 2026-02-25T08:40:16.534 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:16.534 INFO:teuthology.orchestra.run.trial169.stdout:Installed: 2026-02-25T08:40:16.534 INFO:teuthology.orchestra.run.trial169.stdout: ceph-debugsource-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:40:16.534 INFO:teuthology.orchestra.run.trial169.stdout: ceph-osd-crimson-debuginfo-2:20.3.0-5338.g6aeb6975.el9.x86_64 2026-02-25T08:40:16.534 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:16.534 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:16.610 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:16.611 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:16.775 DEBUG:teuthology.orchestra.run.trial169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:40:16.796 INFO:teuthology.orchestra.run.trial169.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:16.797 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:16.797 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:40:16.799 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:16.944 DEBUG:teuthology.orchestra.run.trial183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:40:16.966 INFO:teuthology.orchestra.run.trial183.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:16.966 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:16.966 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:40:16.969 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-25T08:40:16.969 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:40:16.969 DEBUG:teuthology.orchestra.run.trial169:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T08:40:16.997 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:40:16.997 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T08:40:17.031 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-25T08:40:17.031 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:40:17.032 DEBUG:teuthology.orchestra.run.trial169:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T08:40:17.062 DEBUG:teuthology.orchestra.run.trial169:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T08:40:17.126 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:40:17.126 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T08:40:17.153 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T08:40:17.217 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-25T08:40:17.217 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:40:17.217 DEBUG:teuthology.orchestra.run.trial169:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T08:40:17.243 DEBUG:teuthology.orchestra.run.trial169:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T08:40:17.307 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:40:17.307 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T08:40:17.333 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T08:40:17.397 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-25T08:40:17.397 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:40:17.398 DEBUG:teuthology.orchestra.run.trial169:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T08:40:17.423 DEBUG:teuthology.orchestra.run.trial169:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T08:40:17.486 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:40:17.487 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T08:40:17.512 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T08:40:17.577 INFO:teuthology.run_tasks:Running task pexec... 2026-02-25T08:40:17.595 INFO:teuthology.task.pexec:Executing custom commands... 2026-02-25T08:40:17.595 DEBUG:teuthology.orchestra.run.trial169:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-25T08:40:17.596 DEBUG:teuthology.orchestra.run.trial183:> TESTDIR=/home/ubuntu/cephtest bash -s 2026-02-25T08:40:17.599 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial169.front.sepia.ceph.com 2026-02-25T08:40:17.599 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-25T08:40:17.619 INFO:teuthology.task.pexec:Running commands on host ubuntu@trial183.front.sepia.ceph.com 2026-02-25T08:40:17.619 INFO:teuthology.task.pexec:sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson 2026-02-25T08:40:17.650 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:17.670 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:17.670 INFO:teuthology.run_tasks:Running task install... 2026-02-25T08:40:17.684 DEBUG:teuthology.task.install:project ceph 2026-02-25T08:40:17.684 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}} 2026-02-25T08:40:17.684 DEBUG:teuthology.task.install:config {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'} 2026-02-25T08:40:17.684 INFO:teuthology.task.install:Using flavor: debug 2026-02-25T08:40:17.693 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T08:40:17.694 INFO:teuthology.task.install:extra packages: [] 2026-02-25T08:40:17.694 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:40:17.694 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:17.696 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'debug', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'tag': None, 'wait_for_package': False} 2026-02-25T08:40:17.696 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:17.894 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:40:17.894 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:40:17.977 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/ 2026-02-25T08:40:17.977 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5338.g6aeb6975 2026-02-25T08:40:18.061 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:40:18.061 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:40:18.062 DEBUG:teuthology.orchestra.run.trial169:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:40:18.087 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-25T08:40:18.087 DEBUG:teuthology.orchestra.run.trial169:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:40:18.090 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-logstoremanager/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/centos/9/flavors/debug/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-25T08:40:18.090 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:40:18.090 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-25T08:40:18.115 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-25T08:40:18.116 DEBUG:teuthology.orchestra.run.trial183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/6aeb6975bc9a4790a399c9cc70dba9635d2a7819/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-25T08:40:18.151 DEBUG:teuthology.orchestra.run.trial169:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:40:18.181 DEBUG:teuthology.orchestra.run.trial183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-25T08:40:18.215 DEBUG:teuthology.orchestra.run.trial169:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:40:18.245 DEBUG:teuthology.orchestra.run.trial183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-25T08:40:18.271 INFO:teuthology.orchestra.run.trial169.stdout:check_obsoletes = 1 2026-02-25T08:40:18.280 DEBUG:teuthology.orchestra.run.trial169:> sudo yum clean all 2026-02-25T08:40:18.301 INFO:teuthology.orchestra.run.trial183.stdout:check_obsoletes = 1 2026-02-25T08:40:18.311 DEBUG:teuthology.orchestra.run.trial183:> sudo yum clean all 2026-02-25T08:40:18.441 INFO:teuthology.orchestra.run.trial169.stdout:56 files removed 2026-02-25T08:40:18.457 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-radosgw 2026-02-25T08:40:18.472 INFO:teuthology.orchestra.run.trial183.stdout:56 files removed 2026-02-25T08:40:18.487 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-radosgw 2026-02-25T08:40:18.771 INFO:teuthology.orchestra.run.trial169.stdout:ceph packages for x86_64 599 kB/s | 94 kB 00:00 2026-02-25T08:40:18.793 INFO:teuthology.orchestra.run.trial183.stdout:ceph packages for x86_64 635 kB/s | 94 kB 00:00 2026-02-25T08:40:18.924 INFO:teuthology.orchestra.run.trial169.stdout:ceph noarch packages 198 kB/s | 24 kB 00:00 2026-02-25T08:40:18.958 INFO:teuthology.orchestra.run.trial183.stdout:ceph noarch packages 179 kB/s | 24 kB 00:00 2026-02-25T08:40:19.062 INFO:teuthology.orchestra.run.trial169.stdout:ceph source packages 18 kB/s | 2.1 kB 00:00 2026-02-25T08:40:19.096 INFO:teuthology.orchestra.run.trial183.stdout:ceph source packages 18 kB/s | 2.1 kB 00:00 2026-02-25T08:40:19.920 INFO:teuthology.orchestra.run.trial169.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.9 MB 00:00 2026-02-25T08:40:20.312 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 8.9 MB 00:01 2026-02-25T08:40:22.446 INFO:teuthology.orchestra.run.trial169.stdout:CentOS Stream 9 - AppStream 14 MB/s | 27 MB 00:01 2026-02-25T08:40:22.883 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - AppStream 13 MB/s | 27 MB 00:02 2026-02-25T08:40:26.012 INFO:teuthology.orchestra.run.trial169.stdout:CentOS Stream 9 - CRB 6.9 MB/s | 8.0 MB 00:01 2026-02-25T08:40:26.621 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 8.0 MB 00:01 2026-02-25T08:40:26.882 INFO:teuthology.orchestra.run.trial169.stdout:CentOS Stream 9 - Extras packages 182 kB/s | 20 kB 00:00 2026-02-25T08:40:27.460 INFO:teuthology.orchestra.run.trial169.stdout:Extra Packages for Enterprise Linux 39 MB/s | 20 MB 00:00 2026-02-25T08:40:27.473 INFO:teuthology.orchestra.run.trial183.stdout:CentOS Stream 9 - Extras packages 220 kB/s | 20 kB 00:00 2026-02-25T08:40:30.002 INFO:teuthology.orchestra.run.trial183.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 20 MB 00:02 2026-02-25T08:40:30.674 INFO:teuthology.orchestra.run.trial169.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-25T08:40:31.856 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:31.874 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:31.875 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:31.875 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:31.899 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-test 2026-02-25T08:40:32.218 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:32.283 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:32.301 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:32.302 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:32.302 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:32.324 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph 2026-02-25T08:40:32.646 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:32.712 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:32.729 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:32.730 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:32.730 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:32.752 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-base 2026-02-25T08:40:33.072 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:33.137 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:33.155 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:33.156 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:33.156 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:33.178 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install cephadm 2026-02-25T08:40:33.278 INFO:teuthology.orchestra.run.trial183.stdout:lab-extras 2.7 MB/s | 50 kB 00:00 2026-02-25T08:40:33.499 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:33.565 INFO:teuthology.orchestra.run.trial169.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:33.583 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:33.584 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:33.584 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:33.604 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:40:33.928 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:33.993 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:34.011 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:34.011 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:34.012 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:34.034 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-mgr 2026-02-25T08:40:34.355 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:34.420 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:34.438 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:34.439 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:34.439 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:34.461 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:40:34.467 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-radosgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:34.486 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:34.487 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:34.487 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:34.511 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-test 2026-02-25T08:40:34.785 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:34.839 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:01 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:34.850 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:34.868 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:34.869 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:34.869 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:34.891 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:40:34.904 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-test-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:34.923 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:34.923 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:34.923 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:34.946 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph 2026-02-25T08:40:35.214 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:35.262 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:35.279 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:35.296 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:35.297 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:35.297 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:35.320 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:40:35.328 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:35.345 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:35.346 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:35.346 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:35.367 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-base 2026-02-25T08:40:35.637 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:35.689 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:02 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:35.704 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:35.722 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:35.722 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:35.722 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:35.744 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:40:35.756 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-base-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:35.773 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:35.774 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:35.774 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:35.796 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install cephadm 2026-02-25T08:40:36.066 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:36.123 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:36.132 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:36.150 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:36.151 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:36.151 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:36.173 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-osd 2026-02-25T08:40:36.190 INFO:teuthology.orchestra.run.trial183.stdout:Package cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:36.207 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:36.208 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:36.208 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:36.230 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-immutable-object-cache 2026-02-25T08:40:36.492 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:36.554 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:36.558 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:36.575 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:36.576 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:36.576 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:36.598 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-osd-classic 2026-02-25T08:40:36.618 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-immutable-object-cache-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:36.636 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:36.637 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:36.637 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:36.658 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr 2026-02-25T08:40:36.919 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:36.981 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:03 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:36.984 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:37.002 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:37.002 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:37.002 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:37.024 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-fuse 2026-02-25T08:40:37.046 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:37.063 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:37.064 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:37.064 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:37.085 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-dashboard 2026-02-25T08:40:37.346 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:37.411 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:37.412 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:37.429 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:37.429 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:37.429 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:37.452 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install ceph-volume 2026-02-25T08:40:37.478 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-dashboard-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:37.496 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:37.497 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:37.497 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:37.519 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-25T08:40:37.774 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:37.839 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:04 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:37.840 INFO:teuthology.orchestra.run.trial169.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:37.858 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:37.858 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:37.858 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:37.880 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install librados-devel 2026-02-25T08:40:37.905 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:37.923 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:37.923 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:37.923 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:37.944 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-rook 2026-02-25T08:40:38.203 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:38.265 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:38.269 INFO:teuthology.orchestra.run.trial169.stdout:Package librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:38.286 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:38.287 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:38.287 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:38.310 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install libcephfs2 2026-02-25T08:40:38.331 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-rook-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:38.349 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:38.350 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:38.350 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:38.372 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-mgr-cephadm 2026-02-25T08:40:38.629 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:38.695 INFO:teuthology.orchestra.run.trial169.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:38.697 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:05 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:38.713 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:38.713 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:38.713 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:38.735 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install libcephfs-devel 2026-02-25T08:40:38.762 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-mgr-cephadm-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:38.780 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:38.781 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:38.781 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:38.803 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-osd 2026-02-25T08:40:39.056 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:39.121 INFO:teuthology.orchestra.run.trial169.stdout:Package libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:39.135 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:39.139 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:39.140 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:39.140 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:39.162 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install librados2 2026-02-25T08:40:39.202 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-osd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:39.219 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:39.220 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:39.220 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:39.243 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-osd-classic 2026-02-25T08:40:39.484 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:39.549 INFO:teuthology.orchestra.run.trial169.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:39.562 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:39.567 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:39.568 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:39.568 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:39.590 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install librbd1 2026-02-25T08:40:39.629 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-osd-classic-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:39.646 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:39.647 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:39.647 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:39.668 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-fuse 2026-02-25T08:40:39.910 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:39.976 INFO:teuthology.orchestra.run.trial169.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:39.991 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:06 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:39.993 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:39.994 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:39.994 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:40.016 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install python3-rados 2026-02-25T08:40:40.058 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:40.075 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:40.076 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:40.076 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:40.098 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install ceph-volume 2026-02-25T08:40:40.339 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:40.404 INFO:teuthology.orchestra.run.trial169.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:40.421 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:40.422 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:40.423 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:40.423 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:40.445 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install python3-rgw 2026-02-25T08:40:40.487 INFO:teuthology.orchestra.run.trial183.stdout:Package ceph-volume-2:20.3.0-5338.g6aeb6975.el9.noarch is already installed. 2026-02-25T08:40:40.505 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:40.506 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:40.506 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:40.528 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librados-devel 2026-02-25T08:40:40.765 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:40.830 INFO:teuthology.orchestra.run.trial169.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:40.847 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:07 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:40.848 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:40.849 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:40.849 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:40.870 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install python3-cephfs 2026-02-25T08:40:40.912 INFO:teuthology.orchestra.run.trial183.stdout:Package librados-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:40.930 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:40.930 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:40.930 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:40.952 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install libcephfs2 2026-02-25T08:40:41.190 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:41.255 INFO:teuthology.orchestra.run.trial169.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:41.273 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:41.274 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:41.274 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:41.275 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:41.295 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install python3-rbd 2026-02-25T08:40:41.341 INFO:teuthology.orchestra.run.trial183.stdout:Package libcephfs2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:41.358 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:41.359 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:41.359 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:41.381 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install libcephfs-devel 2026-02-25T08:40:41.613 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:41.680 INFO:teuthology.orchestra.run.trial169.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:41.697 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:41.698 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:41.698 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:41.704 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:08 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:41.719 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install rbd-fuse 2026-02-25T08:40:41.770 INFO:teuthology.orchestra.run.trial183.stdout:Package libcephfs-devel-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:41.788 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:41.789 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:41.789 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:41.810 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librados2 2026-02-25T08:40:42.040 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:42.106 INFO:teuthology.orchestra.run.trial169.stdout:Package rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:42.123 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:42.124 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:42.124 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:42.137 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:42.146 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install rbd-mirror 2026-02-25T08:40:42.203 INFO:teuthology.orchestra.run.trial183.stdout:Package librados2-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:42.221 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:42.221 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:42.221 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:42.244 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install librbd1 2026-02-25T08:40:42.466 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:42.532 INFO:teuthology.orchestra.run.trial169.stdout:Package rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:42.549 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:42.550 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:42.550 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:42.568 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:42.572 DEBUG:teuthology.orchestra.run.trial169:> sudo yum -y install rbd-nbd 2026-02-25T08:40:42.634 INFO:teuthology.orchestra.run.trial183.stdout:Package librbd1-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:42.652 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:42.652 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:42.653 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:42.674 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rados 2026-02-25T08:40:42.892 INFO:teuthology.orchestra.run.trial169.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:40:30 AM UTC. 2026-02-25T08:40:42.957 INFO:teuthology.orchestra.run.trial169.stdout:Package rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:42.975 INFO:teuthology.orchestra.run.trial169.stdout:Dependencies resolved. 2026-02-25T08:40:42.975 INFO:teuthology.orchestra.run.trial169.stdout:Nothing to do. 2026-02-25T08:40:42.976 INFO:teuthology.orchestra.run.trial169.stdout:Complete! 2026-02-25T08:40:42.996 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:09 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:42.999 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:43.063 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rados-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:43.081 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:43.082 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:43.082 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:43.104 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rgw 2026-02-25T08:40:43.425 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:43.491 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rgw-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:43.509 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:43.509 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:43.510 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:43.531 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-cephfs 2026-02-25T08:40:43.851 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:10 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:43.917 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-cephfs-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:43.936 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:43.936 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:43.936 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:43.958 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install python3-rbd 2026-02-25T08:40:44.280 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:44.346 INFO:teuthology.orchestra.run.trial183.stdout:Package python3-rbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:44.364 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:44.365 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:44.365 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:44.389 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-fuse 2026-02-25T08:40:44.711 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:11 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:44.776 INFO:teuthology.orchestra.run.trial183.stdout:Package rbd-fuse-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:44.794 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:44.794 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:44.795 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:44.818 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-mirror 2026-02-25T08:40:45.140 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:45.205 INFO:teuthology.orchestra.run.trial183.stdout:Package rbd-mirror-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:45.223 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:45.223 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:45.223 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:45.244 DEBUG:teuthology.orchestra.run.trial183:> sudo yum -y install rbd-nbd 2026-02-25T08:40:45.564 INFO:teuthology.orchestra.run.trial183.stdout:Last metadata expiration check: 0:00:12 ago on Wed 25 Feb 2026 08:40:33 AM UTC. 2026-02-25T08:40:45.630 INFO:teuthology.orchestra.run.trial183.stdout:Package rbd-nbd-2:20.3.0-5338.g6aeb6975.el9.x86_64 is already installed. 2026-02-25T08:40:45.648 INFO:teuthology.orchestra.run.trial183.stdout:Dependencies resolved. 2026-02-25T08:40:45.649 INFO:teuthology.orchestra.run.trial183.stdout:Nothing to do. 2026-02-25T08:40:45.649 INFO:teuthology.orchestra.run.trial183.stdout:Complete! 2026-02-25T08:40:45.672 DEBUG:teuthology.parallel:result is None 2026-02-25T08:40:45.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:45.848 DEBUG:teuthology.orchestra.run.trial169:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:40:45.869 INFO:teuthology.orchestra.run.trial169.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:45.869 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:45.869 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:40:45.871 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=debug&distros=centos%2F9%2Fx86_64&sha1=6aeb6975bc9a4790a399c9cc70dba9635d2a7819 2026-02-25T08:40:46.028 DEBUG:teuthology.orchestra.run.trial183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-25T08:40:46.049 INFO:teuthology.orchestra.run.trial183.stdout:20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:46.050 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5338.g6aeb6975.el9 2026-02-25T08:40:46.050 INFO:teuthology.task.install:The correct ceph version 20.3.0-5338.g6aeb6975 is installed. 2026-02-25T08:40:46.052 INFO:teuthology.task.install.util:Utilities already shipped, skip it... 2026-02-25T08:40:46.052 INFO:teuthology.run_tasks:Running task ceph... 2026-02-25T08:40:46.150 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-25T08:40:46.150 DEBUG:teuthology.orchestra.run.trial169:> sudo chmod 777 /var/log/ceph 2026-02-25T08:40:46.152 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod 777 /var/log/ceph 2026-02-25T08:40:46.177 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-25T08:40:46.178 DEBUG:teuthology.orchestra.run.trial169:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-25T08:40:46.218 DEBUG:teuthology.orchestra.run.trial183:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-25T08:40:46.241 INFO:tasks.ceph:Creating extra log directories... 2026-02-25T08:40:46.241 DEBUG:teuthology.orchestra.run.trial169:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-25T08:40:46.282 DEBUG:teuthology.orchestra.run.trial183:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-25T08:40:46.317 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-25T08:40:46.317 INFO:tasks.ceph:config {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-25T08:40:46.318 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69078', 'branch': 'wip-logstoremanager', 'description': 'crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} thrashers/{default} thrashosds-health workloads/snaps-few-objects-balanced}', 'email': None, 'first_in_suite': False, 'flavor': 'debug', 'job_id': '69078', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'trial', 'name': 'matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial', 'no_nested_subset': False, 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}}, 'owner': 'scheduled_matan@soko04.front.sepia.ceph.com', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'osd.0', 'osd.1', 'client.0', 'node-exporter.a'], ['mgr.x', 'osd.2', 'osd.3', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 869, 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'sleep_before_teardown': 0, 'suite': 'crimson-rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph.git', 'suite_sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d', 'targets': {'trial169.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOxFCEh2riEJZuVhsl0d7Lfw0dtKL2gecErtNAwJLjCJMKGYEp+znTw+NUtgwFx3fzSCf/D+SrhoKzz6wLSPJAM=', 'trial183.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBhkL8/E3SLJ4IZT5lytGCGhBDsJdcorAIMTMxmCMPZ1j8Mnhva1PeYwM7RBGbB9XOqhi8yyi/7PVrz1+3G5FEo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['ceph-osd-crimson', 'ceph-osd-crimson-dbg'], 'rpm': ['ceph-osd-crimson', 'ceph-osd-crimson-debuginfo']}, 'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, {'pexec': {'all': ['sudo update-alternatives --set ceph-osd /usr/bin/ceph-osd-crimson']}}, {'install': None}, {'ceph': {'conf': {'osd': {'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819', 'cluster': 'ceph'}}, {'thrashosds': {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400}}, {'rados': {'balance_reads': True, 'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-25_07:55:35', 'tube': 'trial', 'user': 'matan', 'verbose': False, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699'} 2026-02-25T08:40:46.318 DEBUG:teuthology.orchestra.run.trial169:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-25T08:40:46.347 DEBUG:teuthology.orchestra.run.trial183:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-25T08:40:46.364 DEBUG:teuthology.orchestra.run.trial169:> sudo install -d -m0777 -- /var/run/ceph 2026-02-25T08:40:46.402 DEBUG:teuthology.orchestra.run.trial183:> sudo install -d -m0777 -- /var/run/ceph 2026-02-25T08:40:46.429 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:40:46.429 DEBUG:teuthology.orchestra.run.trial169:> dd if=/scratch_devs of=/dev/stdout 2026-02-25T08:40:46.481 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-25T08:40:46.481 DEBUG:teuthology.orchestra.run.trial169:> stat /dev/vg_nvme/lv_1 2026-02-25T08:40:46.537 INFO:teuthology.orchestra.run.trial169.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-25T08:40:46.537 INFO:teuthology.orchestra.run.trial169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:46.537 INFO:teuthology.orchestra.run.trial169.stdout:Device: 6h/6d Inode: 1291 Links: 1 2026-02-25T08:40:46.537 INFO:teuthology.orchestra.run.trial169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:46.537 INFO:teuthology.orchestra.run.trial169.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:46.537 INFO:teuthology.orchestra.run.trial169.stdout:Access: 2026-02-25 08:24:23.397387274 +0000 2026-02-25T08:40:46.538 INFO:teuthology.orchestra.run.trial169.stdout:Modify: 2026-02-25 08:10:12.737121377 +0000 2026-02-25T08:40:46.538 INFO:teuthology.orchestra.run.trial169.stdout:Change: 2026-02-25 08:10:12.737121377 +0000 2026-02-25T08:40:46.538 INFO:teuthology.orchestra.run.trial169.stdout: Birth: 2026-02-25 08:10:12.737121377 +0000 2026-02-25T08:40:46.538 DEBUG:teuthology.orchestra.run.trial169:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-25T08:40:46.599 INFO:teuthology.orchestra.run.trial169.stderr:1+0 records in 2026-02-25T08:40:46.600 INFO:teuthology.orchestra.run.trial169.stderr:1+0 records out 2026-02-25T08:40:46.600 INFO:teuthology.orchestra.run.trial169.stderr:512 bytes copied, 0.000164445 s, 3.1 MB/s 2026-02-25T08:40:46.601 DEBUG:teuthology.orchestra.run.trial169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-25T08:40:46.659 DEBUG:teuthology.orchestra.run.trial169:> stat /dev/vg_nvme/lv_2 2026-02-25T08:40:46.714 INFO:teuthology.orchestra.run.trial169.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-25T08:40:46.715 INFO:teuthology.orchestra.run.trial169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:46.715 INFO:teuthology.orchestra.run.trial169.stdout:Device: 6h/6d Inode: 1297 Links: 1 2026-02-25T08:40:46.715 INFO:teuthology.orchestra.run.trial169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:46.715 INFO:teuthology.orchestra.run.trial169.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:46.715 INFO:teuthology.orchestra.run.trial169.stdout:Access: 2026-02-25 08:24:23.397387274 +0000 2026-02-25T08:40:46.715 INFO:teuthology.orchestra.run.trial169.stdout:Modify: 2026-02-25 08:10:12.738121337 +0000 2026-02-25T08:40:46.715 INFO:teuthology.orchestra.run.trial169.stdout:Change: 2026-02-25 08:10:12.738121337 +0000 2026-02-25T08:40:46.715 INFO:teuthology.orchestra.run.trial169.stdout: Birth: 2026-02-25 08:10:12.738121337 +0000 2026-02-25T08:40:46.716 DEBUG:teuthology.orchestra.run.trial169:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-25T08:40:46.776 INFO:teuthology.orchestra.run.trial169.stderr:1+0 records in 2026-02-25T08:40:46.777 INFO:teuthology.orchestra.run.trial169.stderr:1+0 records out 2026-02-25T08:40:46.777 INFO:teuthology.orchestra.run.trial169.stderr:512 bytes copied, 0.000141222 s, 3.6 MB/s 2026-02-25T08:40:46.778 DEBUG:teuthology.orchestra.run.trial169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-25T08:40:46.835 DEBUG:teuthology.orchestra.run.trial169:> stat /dev/vg_nvme/lv_3 2026-02-25T08:40:46.890 INFO:teuthology.orchestra.run.trial169.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-25T08:40:46.891 INFO:teuthology.orchestra.run.trial169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:46.891 INFO:teuthology.orchestra.run.trial169.stdout:Device: 6h/6d Inode: 1304 Links: 1 2026-02-25T08:40:46.891 INFO:teuthology.orchestra.run.trial169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:46.891 INFO:teuthology.orchestra.run.trial169.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:46.891 INFO:teuthology.orchestra.run.trial169.stdout:Access: 2026-02-25 08:24:23.398387254 +0000 2026-02-25T08:40:46.891 INFO:teuthology.orchestra.run.trial169.stdout:Modify: 2026-02-25 08:10:12.738121337 +0000 2026-02-25T08:40:46.891 INFO:teuthology.orchestra.run.trial169.stdout:Change: 2026-02-25 08:10:12.738121337 +0000 2026-02-25T08:40:46.891 INFO:teuthology.orchestra.run.trial169.stdout: Birth: 2026-02-25 08:10:12.738121337 +0000 2026-02-25T08:40:46.892 DEBUG:teuthology.orchestra.run.trial169:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-25T08:40:46.953 INFO:teuthology.orchestra.run.trial169.stderr:1+0 records in 2026-02-25T08:40:46.953 INFO:teuthology.orchestra.run.trial169.stderr:1+0 records out 2026-02-25T08:40:46.953 INFO:teuthology.orchestra.run.trial169.stderr:512 bytes copied, 0.00021571 s, 2.4 MB/s 2026-02-25T08:40:46.954 DEBUG:teuthology.orchestra.run.trial169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-25T08:40:47.013 DEBUG:teuthology.orchestra.run.trial169:> stat /dev/vg_nvme/lv_4 2026-02-25T08:40:47.068 INFO:teuthology.orchestra.run.trial169.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-25T08:40:47.069 INFO:teuthology.orchestra.run.trial169.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:47.069 INFO:teuthology.orchestra.run.trial169.stdout:Device: 6h/6d Inode: 1307 Links: 1 2026-02-25T08:40:47.069 INFO:teuthology.orchestra.run.trial169.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:47.069 INFO:teuthology.orchestra.run.trial169.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:47.069 INFO:teuthology.orchestra.run.trial169.stdout:Access: 2026-02-25 08:24:23.398387254 +0000 2026-02-25T08:40:47.069 INFO:teuthology.orchestra.run.trial169.stdout:Modify: 2026-02-25 08:10:12.738121337 +0000 2026-02-25T08:40:47.069 INFO:teuthology.orchestra.run.trial169.stdout:Change: 2026-02-25 08:10:12.738121337 +0000 2026-02-25T08:40:47.069 INFO:teuthology.orchestra.run.trial169.stdout: Birth: 2026-02-25 08:10:12.738121337 +0000 2026-02-25T08:40:47.070 DEBUG:teuthology.orchestra.run.trial169:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-25T08:40:47.132 INFO:teuthology.orchestra.run.trial169.stderr:1+0 records in 2026-02-25T08:40:47.132 INFO:teuthology.orchestra.run.trial169.stderr:1+0 records out 2026-02-25T08:40:47.132 INFO:teuthology.orchestra.run.trial169.stderr:512 bytes copied, 0.000211111 s, 2.4 MB/s 2026-02-25T08:40:47.133 DEBUG:teuthology.orchestra.run.trial169:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-25T08:40:47.191 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:47.191 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:40:47.191 DEBUG:teuthology.orchestra.run.trial183:> dd if=/scratch_devs of=/dev/stdout 2026-02-25T08:40:47.209 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-25T08:40:47.209 DEBUG:teuthology.orchestra.run.trial183:> stat /dev/vg_nvme/lv_1 2026-02-25T08:40:47.265 INFO:teuthology.orchestra.run.trial183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-25T08:40:47.265 INFO:teuthology.orchestra.run.trial183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:47.265 INFO:teuthology.orchestra.run.trial183.stdout:Device: 6h/6d Inode: 1311 Links: 1 2026-02-25T08:40:47.265 INFO:teuthology.orchestra.run.trial183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:47.266 INFO:teuthology.orchestra.run.trial183.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:47.266 INFO:teuthology.orchestra.run.trial183.stdout:Access: 2026-02-25 08:27:56.264513002 +0000 2026-02-25T08:40:47.266 INFO:teuthology.orchestra.run.trial183.stdout:Modify: 2026-02-25 08:12:49.933988495 +0000 2026-02-25T08:40:47.266 INFO:teuthology.orchestra.run.trial183.stdout:Change: 2026-02-25 08:12:49.933988495 +0000 2026-02-25T08:40:47.266 INFO:teuthology.orchestra.run.trial183.stdout: Birth: 2026-02-25 08:12:49.933988495 +0000 2026-02-25T08:40:47.266 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-25T08:40:47.327 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records in 2026-02-25T08:40:47.328 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records out 2026-02-25T08:40:47.328 INFO:teuthology.orchestra.run.trial183.stderr:512 bytes copied, 0.000173954 s, 2.9 MB/s 2026-02-25T08:40:47.329 DEBUG:teuthology.orchestra.run.trial183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-25T08:40:47.386 DEBUG:teuthology.orchestra.run.trial183:> stat /dev/vg_nvme/lv_2 2026-02-25T08:40:47.443 INFO:teuthology.orchestra.run.trial183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-25T08:40:47.443 INFO:teuthology.orchestra.run.trial183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:47.443 INFO:teuthology.orchestra.run.trial183.stdout:Device: 6h/6d Inode: 1291 Links: 1 2026-02-25T08:40:47.443 INFO:teuthology.orchestra.run.trial183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:47.443 INFO:teuthology.orchestra.run.trial183.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:47.443 INFO:teuthology.orchestra.run.trial183.stdout:Access: 2026-02-25 08:27:56.265512978 +0000 2026-02-25T08:40:47.443 INFO:teuthology.orchestra.run.trial183.stdout:Modify: 2026-02-25 08:12:49.931988544 +0000 2026-02-25T08:40:47.444 INFO:teuthology.orchestra.run.trial183.stdout:Change: 2026-02-25 08:12:49.931988544 +0000 2026-02-25T08:40:47.444 INFO:teuthology.orchestra.run.trial183.stdout: Birth: 2026-02-25 08:12:49.931988544 +0000 2026-02-25T08:40:47.444 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-25T08:40:47.506 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records in 2026-02-25T08:40:47.506 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records out 2026-02-25T08:40:47.506 INFO:teuthology.orchestra.run.trial183.stderr:512 bytes copied, 0.000191506 s, 2.7 MB/s 2026-02-25T08:40:47.507 DEBUG:teuthology.orchestra.run.trial183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-25T08:40:47.565 DEBUG:teuthology.orchestra.run.trial183:> stat /dev/vg_nvme/lv_3 2026-02-25T08:40:47.621 INFO:teuthology.orchestra.run.trial183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-25T08:40:47.621 INFO:teuthology.orchestra.run.trial183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:47.621 INFO:teuthology.orchestra.run.trial183.stdout:Device: 6h/6d Inode: 1301 Links: 1 2026-02-25T08:40:47.622 INFO:teuthology.orchestra.run.trial183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:47.622 INFO:teuthology.orchestra.run.trial183.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:47.622 INFO:teuthology.orchestra.run.trial183.stdout:Access: 2026-02-25 08:27:56.265512978 +0000 2026-02-25T08:40:47.622 INFO:teuthology.orchestra.run.trial183.stdout:Modify: 2026-02-25 08:12:49.932988519 +0000 2026-02-25T08:40:47.622 INFO:teuthology.orchestra.run.trial183.stdout:Change: 2026-02-25 08:12:49.932988519 +0000 2026-02-25T08:40:47.622 INFO:teuthology.orchestra.run.trial183.stdout: Birth: 2026-02-25 08:12:49.932988519 +0000 2026-02-25T08:40:47.622 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-25T08:40:47.684 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records in 2026-02-25T08:40:47.684 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records out 2026-02-25T08:40:47.684 INFO:teuthology.orchestra.run.trial183.stderr:512 bytes copied, 0.000137296 s, 3.7 MB/s 2026-02-25T08:40:47.685 DEBUG:teuthology.orchestra.run.trial183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-25T08:40:47.743 DEBUG:teuthology.orchestra.run.trial183:> stat /dev/vg_nvme/lv_4 2026-02-25T08:40:47.800 INFO:teuthology.orchestra.run.trial183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-25T08:40:47.800 INFO:teuthology.orchestra.run.trial183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-25T08:40:47.800 INFO:teuthology.orchestra.run.trial183.stdout:Device: 6h/6d Inode: 1306 Links: 1 2026-02-25T08:40:47.800 INFO:teuthology.orchestra.run.trial183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-25T08:40:47.800 INFO:teuthology.orchestra.run.trial183.stdout:Context: system_u:object_r:device_t:s0 2026-02-25T08:40:47.800 INFO:teuthology.orchestra.run.trial183.stdout:Access: 2026-02-25 08:27:56.265512978 +0000 2026-02-25T08:40:47.801 INFO:teuthology.orchestra.run.trial183.stdout:Modify: 2026-02-25 08:12:49.933988495 +0000 2026-02-25T08:40:47.801 INFO:teuthology.orchestra.run.trial183.stdout:Change: 2026-02-25 08:12:49.933988495 +0000 2026-02-25T08:40:47.801 INFO:teuthology.orchestra.run.trial183.stdout: Birth: 2026-02-25 08:12:49.933988495 +0000 2026-02-25T08:40:47.801 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-25T08:40:47.863 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records in 2026-02-25T08:40:47.863 INFO:teuthology.orchestra.run.trial183.stderr:1+0 records out 2026-02-25T08:40:47.863 INFO:teuthology.orchestra.run.trial183.stderr:512 bytes copied, 0.000144829 s, 3.5 MB/s 2026-02-25T08:40:47.864 DEBUG:teuthology.orchestra.run.trial183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-25T08:40:47.921 INFO:tasks.ceph:osd dev map: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:47.921 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial169.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@trial183.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2026-02-25T08:40:47.921 INFO:tasks.ceph:Generating config... 2026-02-25T08:40:47.924 INFO:tasks.ceph:[osd] debug monc = 20 2026-02-25T08:40:47.924 INFO:tasks.ceph:[osd] crimson cpu num = 4 2026-02-25T08:40:47.924 INFO:tasks.ceph:[osd] crimson osd obc lru size = 10 2026-02-25T08:40:47.924 INFO:tasks.ceph:[osd] debug ms = 20 2026-02-25T08:40:47.924 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-25T08:40:47.925 INFO:tasks.ceph:[osd] debug seastore = 20 2026-02-25T08:40:47.925 INFO:tasks.ceph:[osd] debug seastore backref = 20 2026-02-25T08:40:47.925 INFO:tasks.ceph:[osd] debug seastore cache = 20 2026-02-25T08:40:47.925 INFO:tasks.ceph:[osd] debug seastore cleaner = 20 2026-02-25T08:40:47.925 INFO:tasks.ceph:[osd] debug seastore device = 20 2026-02-25T08:40:47.925 INFO:tasks.ceph:[osd] debug seastore epm = 20 2026-02-25T08:40:47.925 INFO:tasks.ceph:[osd] debug seastore fixedkv tree = 20 2026-02-25T08:40:47.925 INFO:tasks.ceph:[osd] debug seastore journal = 20 2026-02-25T08:40:47.925 INFO:tasks.ceph:[osd] debug seastore lba = 20 2026-02-25T08:40:47.925 INFO:tasks.ceph:[osd] debug seastore odata = 20 2026-02-25T08:40:47.925 INFO:tasks.ceph:[osd] debug seastore omap = 20 2026-02-25T08:40:47.925 INFO:tasks.ceph:[osd] debug seastore onode = 20 2026-02-25T08:40:47.925 INFO:tasks.ceph:[osd] debug seastore t = 20 2026-02-25T08:40:47.926 INFO:tasks.ceph:[osd] debug seastore tm = 20 2026-02-25T08:40:47.926 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2026-02-25T08:40:47.926 INFO:tasks.ceph:[osd] osd debug reject backfill probability = 0.3 2026-02-25T08:40:47.926 INFO:tasks.ceph:[osd] osd delete sleep = 1 2026-02-25T08:40:47.926 INFO:tasks.ceph:[osd] osd max backfills = 3 2026-02-25T08:40:47.926 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2026-02-25T08:40:47.926 INFO:tasks.ceph:[osd] osd objectstore = seastore 2026-02-25T08:40:47.926 INFO:tasks.ceph:[osd] osd scrub max interval = 120 2026-02-25T08:40:47.926 INFO:tasks.ceph:[osd] osd scrub min interval = 60 2026-02-25T08:40:47.926 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2026-02-25T08:40:47.926 INFO:tasks.ceph:[osd] osd snap trim sleep = 2 2026-02-25T08:40:47.926 INFO:tasks.ceph:[osd] osd target transaction size = 5 2026-02-25T08:40:47.927 INFO:tasks.ceph:[osd] seastore cachepin size pershard = 64M 2026-02-25T08:40:47.927 INFO:tasks.ceph:[osd] seastore main device type = RANDOM_BLOCK_SSD 2026-02-25T08:40:47.927 INFO:tasks.ceph:[osd] seastore max concurrent transactions = 8 2026-02-25T08:40:47.927 INFO:tasks.ceph:[global] enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:40:47.927 INFO:tasks.ceph:[global] ms client mode = crc 2026-02-25T08:40:47.927 INFO:tasks.ceph:[global] ms cluster mode = crc 2026-02-25T08:40:47.927 INFO:tasks.ceph:[global] ms mon client mode = crc 2026-02-25T08:40:47.927 INFO:tasks.ceph:[global] ms mon cluster mode = crc 2026-02-25T08:40:47.927 INFO:tasks.ceph:[global] ms mon service mode = crc 2026-02-25T08:40:47.927 INFO:tasks.ceph:[global] ms service mode = crc 2026-02-25T08:40:47.927 INFO:tasks.ceph:[global] osd_pool_default_min_size = 2 2026-02-25T08:40:47.927 INFO:tasks.ceph:[global] osd_pool_default_size = 3 2026-02-25T08:40:47.928 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-25T08:40:47.928 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-25T08:40:47.928 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-25T08:40:47.928 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-25T08:40:47.928 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-25T08:40:47.928 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2026-02-25T08:40:47.928 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2026-02-25T08:40:47.928 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2026-02-25T08:40:47.928 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2026-02-25T08:40:47.928 INFO:tasks.ceph:[mon] osd pool default crimson = True 2026-02-25T08:40:47.928 INFO:tasks.ceph:[mon] paxos service trim min = 10 2026-02-25T08:40:47.929 INFO:tasks.ceph:Setting up mon.a... 2026-02-25T08:40:47.929 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-25T08:40:47.971 INFO:teuthology.orchestra.run.trial169.stdout:creating /etc/ceph/ceph.keyring 2026-02-25T08:40:47.974 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-25T08:40:48.057 DEBUG:teuthology.orchestra.run.trial169:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:40:48.121 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.169')] 2026-02-25T08:40:48.122 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.169', 'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': 120, 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug monc': 20, 'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}} 2026-02-25T08:40:48.122 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:40:48.122 DEBUG:teuthology.orchestra.run.trial169:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T08:40:48.178 DEBUG:teuthology.orchestra.run.trial169:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.169 --print /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:48.258 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:40:48.256+0000 7f8401c91f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:48.258 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:40:48.256+0000 7f8401c91f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:48.259 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:40:48.256+0000 7f8401c91f40 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:40:48.260 INFO:teuthology.orchestra.run.trial169.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:48.260 INFO:teuthology.orchestra.run.trial169.stdout:monmaptool: generated fsid a3da792f-4239-4b1a-ad7d-525e6c05ee6a 2026-02-25T08:40:48.260 INFO:teuthology.orchestra.run.trial169.stdout:setting min_mon_release = tentacle 2026-02-25T08:40:48.260 INFO:teuthology.orchestra.run.trial169.stdout:epoch 0 2026-02-25T08:40:48.260 INFO:teuthology.orchestra.run.trial169.stdout:fsid a3da792f-4239-4b1a-ad7d-525e6c05ee6a 2026-02-25T08:40:48.260 INFO:teuthology.orchestra.run.trial169.stdout:last_changed 2026-02-25T08:40:48.257929+0000 2026-02-25T08:40:48.260 INFO:teuthology.orchestra.run.trial169.stdout:created 2026-02-25T08:40:48.257929+0000 2026-02-25T08:40:48.261 INFO:teuthology.orchestra.run.trial169.stdout:min_mon_release 20 (tentacle) 2026-02-25T08:40:48.261 INFO:teuthology.orchestra.run.trial169.stdout:election_strategy: 1 2026-02-25T08:40:48.261 INFO:teuthology.orchestra.run.trial169.stdout:0: [v2:10.20.193.169:3300/0,v1:10.20.193.169:6789/0] mon.a 2026-02-25T08:40:48.261 INFO:teuthology.orchestra.run.trial169.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (1 monitors) 2026-02-25T08:40:48.262 DEBUG:teuthology.orchestra.run.trial169:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-25T08:40:48.319 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a3da792f-4239-4b1a-ad7d-525e6c05ee6a... 2026-02-25T08:40:48.321 DEBUG:teuthology.orchestra.run.trial169:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-25T08:40:48.361 DEBUG:teuthology.orchestra.run.trial183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-25T08:40:48.401 INFO:teuthology.orchestra.run.trial169.stdout:[global] 2026-02-25T08:40:48.401 INFO:teuthology.orchestra.run.trial169.stdout: chdir = "" 2026-02-25T08:40:48.401 INFO:teuthology.orchestra.run.trial169.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-25T08:40:48.401 INFO:teuthology.orchestra.run.trial169.stdout: auth supported = cephx 2026-02-25T08:40:48.401 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.402 INFO:teuthology.orchestra.run.trial169.stdout: filestore xattr use omap = true 2026-02-25T08:40:48.402 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.402 INFO:teuthology.orchestra.run.trial169.stdout: mon clock drift allowed = 1.000 2026-02-25T08:40:48.402 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.402 INFO:teuthology.orchestra.run.trial169.stdout: osd crush chooseleaf type = 0 2026-02-25T08:40:48.402 INFO:teuthology.orchestra.run.trial169.stdout: auth debug = true 2026-02-25T08:40:48.402 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.402 INFO:teuthology.orchestra.run.trial169.stdout: ms die on old message = true 2026-02-25T08:40:48.402 INFO:teuthology.orchestra.run.trial169.stdout: ms die on bug = true 2026-02-25T08:40:48.402 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.402 INFO:teuthology.orchestra.run.trial169.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-25T08:40:48.402 INFO:teuthology.orchestra.run.trial169.stdout: mon pg warn max object skew = 0 2026-02-25T08:40:48.403 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.403 INFO:teuthology.orchestra.run.trial169.stdout: # disable pg_autoscaler by default for new pools 2026-02-25T08:40:48.403 INFO:teuthology.orchestra.run.trial169.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-25T08:40:48.403 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.403 INFO:teuthology.orchestra.run.trial169.stdout: osd pool default size = 2 2026-02-25T08:40:48.403 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.403 INFO:teuthology.orchestra.run.trial169.stdout: mon osd allow primary affinity = true 2026-02-25T08:40:48.403 INFO:teuthology.orchestra.run.trial169.stdout: mon osd allow pg remap = true 2026-02-25T08:40:48.403 INFO:teuthology.orchestra.run.trial169.stdout: mon warn on legacy crush tunables = false 2026-02-25T08:40:48.403 INFO:teuthology.orchestra.run.trial169.stdout: mon warn on crush straw calc version zero = false 2026-02-25T08:40:48.403 INFO:teuthology.orchestra.run.trial169.stdout: mon warn on no sortbitwise = false 2026-02-25T08:40:48.403 INFO:teuthology.orchestra.run.trial169.stdout: mon warn on osd down out interval zero = false 2026-02-25T08:40:48.403 INFO:teuthology.orchestra.run.trial169.stdout: mon warn on too few osds = false 2026-02-25T08:40:48.404 INFO:teuthology.orchestra.run.trial169.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-25T08:40:48.404 INFO:teuthology.orchestra.run.trial169.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-25T08:40:48.404 INFO:teuthology.orchestra.run.trial169.stdout: mon_allow_pool_size_one = true 2026-02-25T08:40:48.404 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.404 INFO:teuthology.orchestra.run.trial169.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-25T08:40:48.404 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.404 INFO:teuthology.orchestra.run.trial169.stdout: osd default data pool replay window = 5 2026-02-25T08:40:48.404 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.404 INFO:teuthology.orchestra.run.trial169.stdout: mon allow pool delete = true 2026-02-25T08:40:48.404 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.404 INFO:teuthology.orchestra.run.trial169.stdout: mon cluster log file level = debug 2026-02-25T08:40:48.405 INFO:teuthology.orchestra.run.trial169.stdout: debug asserts on shutdown = true 2026-02-25T08:40:48.405 INFO:teuthology.orchestra.run.trial169.stdout: mon health detail to clog = false 2026-02-25T08:40:48.405 INFO:teuthology.orchestra.run.trial169.stdout: mon host = 10.20.193.169 2026-02-25T08:40:48.405 INFO:teuthology.orchestra.run.trial169.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:40:48.405 INFO:teuthology.orchestra.run.trial169.stdout: ms client mode = crc 2026-02-25T08:40:48.405 INFO:teuthology.orchestra.run.trial169.stdout: ms cluster mode = crc 2026-02-25T08:40:48.405 INFO:teuthology.orchestra.run.trial169.stdout: ms mon client mode = crc 2026-02-25T08:40:48.405 INFO:teuthology.orchestra.run.trial169.stdout: ms mon cluster mode = crc 2026-02-25T08:40:48.405 INFO:teuthology.orchestra.run.trial169.stdout: ms mon service mode = crc 2026-02-25T08:40:48.405 INFO:teuthology.orchestra.run.trial169.stdout: ms service mode = crc 2026-02-25T08:40:48.405 INFO:teuthology.orchestra.run.trial169.stdout: osd_pool_default_min_size = 2 2026-02-25T08:40:48.405 INFO:teuthology.orchestra.run.trial169.stdout: osd_pool_default_size = 3 2026-02-25T08:40:48.405 INFO:teuthology.orchestra.run.trial169.stdout: fsid = a3da792f-4239-4b1a-ad7d-525e6c05ee6a 2026-02-25T08:40:48.406 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.406 INFO:teuthology.orchestra.run.trial169.stdout:[osd] 2026-02-25T08:40:48.406 INFO:teuthology.orchestra.run.trial169.stdout: osd journal size = 100 2026-02-25T08:40:48.406 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.406 INFO:teuthology.orchestra.run.trial169.stdout: osd scrub load threshold = 5.0 2026-02-25T08:40:48.406 INFO:teuthology.orchestra.run.trial169.stdout: osd scrub max interval = 120 2026-02-25T08:40:48.406 INFO:teuthology.orchestra.run.trial169.stdout: osd mclock profile = high_recovery_ops 2026-02-25T08:40:48.406 INFO:teuthology.orchestra.run.trial169.stdout: osd mclock skip benchmark = true 2026-02-25T08:40:48.406 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.406 INFO:teuthology.orchestra.run.trial169.stdout: osd recover clone overlap = true 2026-02-25T08:40:48.406 INFO:teuthology.orchestra.run.trial169.stdout: osd recovery max chunk = 1048576 2026-02-25T08:40:48.406 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.407 INFO:teuthology.orchestra.run.trial169.stdout: osd debug shutdown = true 2026-02-25T08:40:48.407 INFO:teuthology.orchestra.run.trial169.stdout: osd debug op order = true 2026-02-25T08:40:48.407 INFO:teuthology.orchestra.run.trial169.stdout: osd debug verify stray on activate = true 2026-02-25T08:40:48.407 INFO:teuthology.orchestra.run.trial169.stdout: osd debug trim objects = true 2026-02-25T08:40:48.407 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.407 INFO:teuthology.orchestra.run.trial169.stdout: osd open classes on start = true 2026-02-25T08:40:48.407 INFO:teuthology.orchestra.run.trial169.stdout: osd debug pg log writeout = true 2026-02-25T08:40:48.407 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.407 INFO:teuthology.orchestra.run.trial169.stdout: osd deep scrub update digest min age = 30 2026-02-25T08:40:48.407 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.407 INFO:teuthology.orchestra.run.trial169.stdout: osd map max advance = 10 2026-02-25T08:40:48.407 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.407 INFO:teuthology.orchestra.run.trial169.stdout: journal zero on create = true 2026-02-25T08:40:48.408 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.408 INFO:teuthology.orchestra.run.trial169.stdout: filestore ondisk finisher threads = 3 2026-02-25T08:40:48.408 INFO:teuthology.orchestra.run.trial169.stdout: filestore apply finisher threads = 3 2026-02-25T08:40:48.408 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.408 INFO:teuthology.orchestra.run.trial169.stdout: bdev debug aio = true 2026-02-25T08:40:48.408 INFO:teuthology.orchestra.run.trial169.stdout: osd debug misdirected ops = true 2026-02-25T08:40:48.408 INFO:teuthology.orchestra.run.trial169.stdout: debug monc = 20 2026-02-25T08:40:48.408 INFO:teuthology.orchestra.run.trial169.stdout: crimson cpu num = 4 2026-02-25T08:40:48.408 INFO:teuthology.orchestra.run.trial169.stdout: crimson osd obc lru size = 10 2026-02-25T08:40:48.408 INFO:teuthology.orchestra.run.trial169.stdout: debug ms = 20 2026-02-25T08:40:48.408 INFO:teuthology.orchestra.run.trial169.stdout: debug osd = 20 2026-02-25T08:40:48.408 INFO:teuthology.orchestra.run.trial169.stdout: debug seastore = 20 2026-02-25T08:40:48.408 INFO:teuthology.orchestra.run.trial169.stdout: debug seastore backref = 20 2026-02-25T08:40:48.409 INFO:teuthology.orchestra.run.trial169.stdout: debug seastore cache = 20 2026-02-25T08:40:48.409 INFO:teuthology.orchestra.run.trial169.stdout: debug seastore cleaner = 20 2026-02-25T08:40:48.409 INFO:teuthology.orchestra.run.trial169.stdout: debug seastore device = 20 2026-02-25T08:40:48.409 INFO:teuthology.orchestra.run.trial169.stdout: debug seastore epm = 20 2026-02-25T08:40:48.409 INFO:teuthology.orchestra.run.trial169.stdout: debug seastore fixedkv tree = 20 2026-02-25T08:40:48.409 INFO:teuthology.orchestra.run.trial169.stdout: debug seastore journal = 20 2026-02-25T08:40:48.409 INFO:teuthology.orchestra.run.trial169.stdout: debug seastore lba = 20 2026-02-25T08:40:48.409 INFO:teuthology.orchestra.run.trial169.stdout: debug seastore odata = 20 2026-02-25T08:40:48.409 INFO:teuthology.orchestra.run.trial169.stdout: debug seastore omap = 20 2026-02-25T08:40:48.409 INFO:teuthology.orchestra.run.trial169.stdout: debug seastore onode = 20 2026-02-25T08:40:48.409 INFO:teuthology.orchestra.run.trial169.stdout: debug seastore t = 20 2026-02-25T08:40:48.409 INFO:teuthology.orchestra.run.trial169.stdout: debug seastore tm = 20 2026-02-25T08:40:48.409 INFO:teuthology.orchestra.run.trial169.stdout: osd blocked scrub grace period = 3600 2026-02-25T08:40:48.409 INFO:teuthology.orchestra.run.trial169.stdout: osd debug reject backfill probability = 0.3 2026-02-25T08:40:48.410 INFO:teuthology.orchestra.run.trial169.stdout: osd delete sleep = 1 2026-02-25T08:40:48.410 INFO:teuthology.orchestra.run.trial169.stdout: osd max backfills = 3 2026-02-25T08:40:48.410 INFO:teuthology.orchestra.run.trial169.stdout: osd max markdown count = 1000 2026-02-25T08:40:48.410 INFO:teuthology.orchestra.run.trial169.stdout: osd objectstore = seastore 2026-02-25T08:40:48.410 INFO:teuthology.orchestra.run.trial169.stdout: osd scrub min interval = 60 2026-02-25T08:40:48.410 INFO:teuthology.orchestra.run.trial169.stdout: osd shutdown pgref assert = True 2026-02-25T08:40:48.410 INFO:teuthology.orchestra.run.trial169.stdout: osd snap trim sleep = 2 2026-02-25T08:40:48.410 INFO:teuthology.orchestra.run.trial169.stdout: osd target transaction size = 5 2026-02-25T08:40:48.410 INFO:teuthology.orchestra.run.trial169.stdout: seastore cachepin size pershard = 64M 2026-02-25T08:40:48.410 INFO:teuthology.orchestra.run.trial169.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-25T08:40:48.410 INFO:teuthology.orchestra.run.trial169.stdout: seastore max concurrent transactions = 8 2026-02-25T08:40:48.410 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.410 INFO:teuthology.orchestra.run.trial169.stdout:[mgr] 2026-02-25T08:40:48.411 INFO:teuthology.orchestra.run.trial169.stdout: debug ms = 1 2026-02-25T08:40:48.411 INFO:teuthology.orchestra.run.trial169.stdout: debug mgr = 20 2026-02-25T08:40:48.411 INFO:teuthology.orchestra.run.trial169.stdout: debug mon = 20 2026-02-25T08:40:48.411 INFO:teuthology.orchestra.run.trial169.stdout: debug auth = 20 2026-02-25T08:40:48.411 INFO:teuthology.orchestra.run.trial169.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:48.411 INFO:teuthology.orchestra.run.trial169.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:48.411 INFO:teuthology.orchestra.run.trial169.stdout: mgr/telemetry/nag = false 2026-02-25T08:40:48.411 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.411 INFO:teuthology.orchestra.run.trial169.stdout:[mon] 2026-02-25T08:40:48.411 INFO:teuthology.orchestra.run.trial169.stdout: debug ms = 1 2026-02-25T08:40:48.411 INFO:teuthology.orchestra.run.trial169.stdout: debug mon = 20 2026-02-25T08:40:48.411 INFO:teuthology.orchestra.run.trial169.stdout: debug paxos = 20 2026-02-25T08:40:48.411 INFO:teuthology.orchestra.run.trial169.stdout: debug auth = 20 2026-02-25T08:40:48.412 INFO:teuthology.orchestra.run.trial169.stdout: mon data avail warn = 5 2026-02-25T08:40:48.412 INFO:teuthology.orchestra.run.trial169.stdout: mon mgr mkfs grace = 240 2026-02-25T08:40:48.412 INFO:teuthology.orchestra.run.trial169.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:48.412 INFO:teuthology.orchestra.run.trial169.stdout: mon osd reporter subtree level = osd 2026-02-25T08:40:48.412 INFO:teuthology.orchestra.run.trial169.stdout: mon osd prime pg temp = true 2026-02-25T08:40:48.412 INFO:teuthology.orchestra.run.trial169.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:48.412 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.412 INFO:teuthology.orchestra.run.trial169.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-25T08:40:48.412 INFO:teuthology.orchestra.run.trial169.stdout: auth mon ticket ttl = 660 # 11m 2026-02-25T08:40:48.412 INFO:teuthology.orchestra.run.trial169.stdout: auth service ticket ttl = 240 # 4m 2026-02-25T08:40:48.412 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.412 INFO:teuthology.orchestra.run.trial169.stdout: # don't complain about insecure global_id in the test suite 2026-02-25T08:40:48.412 INFO:teuthology.orchestra.run.trial169.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-25T08:40:48.413 INFO:teuthology.orchestra.run.trial169.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-25T08:40:48.413 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.413 INFO:teuthology.orchestra.run.trial169.stdout: # 1m isn't quite enough 2026-02-25T08:40:48.413 INFO:teuthology.orchestra.run.trial169.stdout: mon_down_mkfs_grace = 2m 2026-02-25T08:40:48.413 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.413 INFO:teuthology.orchestra.run.trial169.stdout: mon_warn_on_filestore_osds = false 2026-02-25T08:40:48.413 INFO:teuthology.orchestra.run.trial169.stdout: mon min osdmap epochs = 50 2026-02-25T08:40:48.413 INFO:teuthology.orchestra.run.trial169.stdout: mon osdmap full prune interval = 2 2026-02-25T08:40:48.413 INFO:teuthology.orchestra.run.trial169.stdout: mon osdmap full prune min = 15 2026-02-25T08:40:48.413 INFO:teuthology.orchestra.run.trial169.stdout: mon osdmap full prune txsize = 2 2026-02-25T08:40:48.413 INFO:teuthology.orchestra.run.trial169.stdout: osd pool default crimson = True 2026-02-25T08:40:48.413 INFO:teuthology.orchestra.run.trial169.stdout: paxos service trim min = 10 2026-02-25T08:40:48.413 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:40:48.414 INFO:teuthology.orchestra.run.trial169.stdout:[client] 2026-02-25T08:40:48.414 INFO:teuthology.orchestra.run.trial169.stdout: rgw cache enabled = true 2026-02-25T08:40:48.414 INFO:teuthology.orchestra.run.trial169.stdout: rgw enable ops log = true 2026-02-25T08:40:48.414 INFO:teuthology.orchestra.run.trial169.stdout: rgw enable usage log = true 2026-02-25T08:40:48.414 INFO:teuthology.orchestra.run.trial169.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-25T08:40:48.414 INFO:teuthology.orchestra.run.trial169.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-25T08:40:48.414 INFO:teuthology.orchestra.run.trial169.stdout:[mon.a] 2026-02-25T08:40:48.416 INFO:teuthology.orchestra.run.trial183.stdout:[global] 2026-02-25T08:40:48.416 INFO:teuthology.orchestra.run.trial183.stdout: chdir = "" 2026-02-25T08:40:48.416 INFO:teuthology.orchestra.run.trial183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-25T08:40:48.416 INFO:teuthology.orchestra.run.trial183.stdout: auth supported = cephx 2026-02-25T08:40:48.416 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.416 INFO:teuthology.orchestra.run.trial183.stdout: filestore xattr use omap = true 2026-02-25T08:40:48.417 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.417 INFO:teuthology.orchestra.run.trial183.stdout: mon clock drift allowed = 1.000 2026-02-25T08:40:48.417 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.417 INFO:teuthology.orchestra.run.trial183.stdout: osd crush chooseleaf type = 0 2026-02-25T08:40:48.417 INFO:teuthology.orchestra.run.trial183.stdout: auth debug = true 2026-02-25T08:40:48.417 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.417 INFO:teuthology.orchestra.run.trial183.stdout: ms die on old message = true 2026-02-25T08:40:48.417 INFO:teuthology.orchestra.run.trial183.stdout: ms die on bug = true 2026-02-25T08:40:48.417 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.417 INFO:teuthology.orchestra.run.trial183.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-25T08:40:48.417 INFO:teuthology.orchestra.run.trial183.stdout: mon pg warn max object skew = 0 2026-02-25T08:40:48.418 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.418 INFO:teuthology.orchestra.run.trial183.stdout: # disable pg_autoscaler by default for new pools 2026-02-25T08:40:48.418 INFO:teuthology.orchestra.run.trial183.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-25T08:40:48.418 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.418 INFO:teuthology.orchestra.run.trial183.stdout: osd pool default size = 2 2026-02-25T08:40:48.418 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.418 INFO:teuthology.orchestra.run.trial183.stdout: mon osd allow primary affinity = true 2026-02-25T08:40:48.418 INFO:teuthology.orchestra.run.trial183.stdout: mon osd allow pg remap = true 2026-02-25T08:40:48.418 INFO:teuthology.orchestra.run.trial183.stdout: mon warn on legacy crush tunables = false 2026-02-25T08:40:48.418 INFO:teuthology.orchestra.run.trial183.stdout: mon warn on crush straw calc version zero = false 2026-02-25T08:40:48.418 INFO:teuthology.orchestra.run.trial183.stdout: mon warn on no sortbitwise = false 2026-02-25T08:40:48.418 INFO:teuthology.orchestra.run.trial183.stdout: mon warn on osd down out interval zero = false 2026-02-25T08:40:48.418 INFO:teuthology.orchestra.run.trial183.stdout: mon warn on too few osds = false 2026-02-25T08:40:48.419 INFO:teuthology.orchestra.run.trial183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-25T08:40:48.419 INFO:teuthology.orchestra.run.trial183.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-25T08:40:48.419 INFO:teuthology.orchestra.run.trial183.stdout: mon_allow_pool_size_one = true 2026-02-25T08:40:48.419 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.419 INFO:teuthology.orchestra.run.trial183.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-25T08:40:48.419 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.419 INFO:teuthology.orchestra.run.trial183.stdout: osd default data pool replay window = 5 2026-02-25T08:40:48.419 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.419 INFO:teuthology.orchestra.run.trial183.stdout: mon allow pool delete = true 2026-02-25T08:40:48.419 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.419 INFO:teuthology.orchestra.run.trial183.stdout: mon cluster log file level = debug 2026-02-25T08:40:48.419 INFO:teuthology.orchestra.run.trial183.stdout: debug asserts on shutdown = true 2026-02-25T08:40:48.420 INFO:teuthology.orchestra.run.trial183.stdout: mon health detail to clog = false 2026-02-25T08:40:48.420 INFO:teuthology.orchestra.run.trial183.stdout: mon host = 10.20.193.169 2026-02-25T08:40:48.420 INFO:teuthology.orchestra.run.trial183.stdout: enable experimental unrecoverable data corrupting features = crimson 2026-02-25T08:40:48.420 INFO:teuthology.orchestra.run.trial183.stdout: ms client mode = crc 2026-02-25T08:40:48.420 INFO:teuthology.orchestra.run.trial183.stdout: ms cluster mode = crc 2026-02-25T08:40:48.420 INFO:teuthology.orchestra.run.trial183.stdout: ms mon client mode = crc 2026-02-25T08:40:48.420 INFO:teuthology.orchestra.run.trial183.stdout: ms mon cluster mode = crc 2026-02-25T08:40:48.420 INFO:teuthology.orchestra.run.trial183.stdout: ms mon service mode = crc 2026-02-25T08:40:48.420 INFO:teuthology.orchestra.run.trial183.stdout: ms service mode = crc 2026-02-25T08:40:48.420 INFO:teuthology.orchestra.run.trial183.stdout: osd_pool_default_min_size = 2 2026-02-25T08:40:48.420 INFO:teuthology.orchestra.run.trial183.stdout: osd_pool_default_size = 3 2026-02-25T08:40:48.420 INFO:teuthology.orchestra.run.trial183.stdout: fsid = a3da792f-4239-4b1a-ad7d-525e6c05ee6a 2026-02-25T08:40:48.420 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.421 INFO:teuthology.orchestra.run.trial183.stdout:[osd] 2026-02-25T08:40:48.421 INFO:teuthology.orchestra.run.trial183.stdout: osd journal size = 100 2026-02-25T08:40:48.421 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.421 INFO:teuthology.orchestra.run.trial183.stdout: osd scrub load threshold = 5.0 2026-02-25T08:40:48.421 INFO:teuthology.orchestra.run.trial183.stdout: osd scrub max interval = 120 2026-02-25T08:40:48.421 INFO:teuthology.orchestra.run.trial183.stdout: osd mclock profile = high_recovery_ops 2026-02-25T08:40:48.421 INFO:teuthology.orchestra.run.trial183.stdout: osd mclock skip benchmark = true 2026-02-25T08:40:48.421 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.421 INFO:teuthology.orchestra.run.trial183.stdout: osd recover clone overlap = true 2026-02-25T08:40:48.421 INFO:teuthology.orchestra.run.trial183.stdout: osd recovery max chunk = 1048576 2026-02-25T08:40:48.421 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.421 INFO:teuthology.orchestra.run.trial183.stdout: osd debug shutdown = true 2026-02-25T08:40:48.421 INFO:teuthology.orchestra.run.trial183.stdout: osd debug op order = true 2026-02-25T08:40:48.422 INFO:teuthology.orchestra.run.trial183.stdout: osd debug verify stray on activate = true 2026-02-25T08:40:48.422 INFO:teuthology.orchestra.run.trial183.stdout: osd debug trim objects = true 2026-02-25T08:40:48.422 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.422 INFO:teuthology.orchestra.run.trial183.stdout: osd open classes on start = true 2026-02-25T08:40:48.422 INFO:teuthology.orchestra.run.trial183.stdout: osd debug pg log writeout = true 2026-02-25T08:40:48.422 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.422 INFO:teuthology.orchestra.run.trial183.stdout: osd deep scrub update digest min age = 30 2026-02-25T08:40:48.422 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.422 INFO:teuthology.orchestra.run.trial183.stdout: osd map max advance = 10 2026-02-25T08:40:48.422 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.422 INFO:teuthology.orchestra.run.trial183.stdout: journal zero on create = true 2026-02-25T08:40:48.422 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.423 INFO:teuthology.orchestra.run.trial183.stdout: filestore ondisk finisher threads = 3 2026-02-25T08:40:48.423 INFO:teuthology.orchestra.run.trial183.stdout: filestore apply finisher threads = 3 2026-02-25T08:40:48.423 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.423 INFO:teuthology.orchestra.run.trial183.stdout: bdev debug aio = true 2026-02-25T08:40:48.423 INFO:teuthology.orchestra.run.trial183.stdout: osd debug misdirected ops = true 2026-02-25T08:40:48.423 INFO:teuthology.orchestra.run.trial183.stdout: debug monc = 20 2026-02-25T08:40:48.423 INFO:teuthology.orchestra.run.trial183.stdout: crimson cpu num = 4 2026-02-25T08:40:48.423 INFO:teuthology.orchestra.run.trial183.stdout: crimson osd obc lru size = 10 2026-02-25T08:40:48.423 INFO:teuthology.orchestra.run.trial183.stdout: debug ms = 20 2026-02-25T08:40:48.423 INFO:teuthology.orchestra.run.trial183.stdout: debug osd = 20 2026-02-25T08:40:48.423 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore = 20 2026-02-25T08:40:48.423 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore backref = 20 2026-02-25T08:40:48.423 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore cache = 20 2026-02-25T08:40:48.424 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore cleaner = 20 2026-02-25T08:40:48.424 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore device = 20 2026-02-25T08:40:48.424 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore epm = 20 2026-02-25T08:40:48.424 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore fixedkv tree = 20 2026-02-25T08:40:48.424 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore journal = 20 2026-02-25T08:40:48.424 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore lba = 20 2026-02-25T08:40:48.424 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore odata = 20 2026-02-25T08:40:48.424 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore omap = 20 2026-02-25T08:40:48.424 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore onode = 20 2026-02-25T08:40:48.424 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore t = 20 2026-02-25T08:40:48.424 INFO:teuthology.orchestra.run.trial183.stdout: debug seastore tm = 20 2026-02-25T08:40:48.424 INFO:teuthology.orchestra.run.trial183.stdout: osd blocked scrub grace period = 3600 2026-02-25T08:40:48.424 INFO:teuthology.orchestra.run.trial183.stdout: osd debug reject backfill probability = 0.3 2026-02-25T08:40:48.425 INFO:teuthology.orchestra.run.trial183.stdout: osd delete sleep = 1 2026-02-25T08:40:48.425 INFO:teuthology.orchestra.run.trial183.stdout: osd max backfills = 3 2026-02-25T08:40:48.425 INFO:teuthology.orchestra.run.trial183.stdout: osd max markdown count = 1000 2026-02-25T08:40:48.425 INFO:teuthology.orchestra.run.trial183.stdout: osd objectstore = seastore 2026-02-25T08:40:48.425 INFO:teuthology.orchestra.run.trial183.stdout: osd scrub min interval = 60 2026-02-25T08:40:48.425 INFO:teuthology.orchestra.run.trial183.stdout: osd shutdown pgref assert = True 2026-02-25T08:40:48.425 INFO:teuthology.orchestra.run.trial183.stdout: osd snap trim sleep = 2 2026-02-25T08:40:48.425 INFO:teuthology.orchestra.run.trial183.stdout: osd target transaction size = 5 2026-02-25T08:40:48.425 INFO:teuthology.orchestra.run.trial183.stdout: seastore cachepin size pershard = 64M 2026-02-25T08:40:48.425 INFO:teuthology.orchestra.run.trial183.stdout: seastore main device type = RANDOM_BLOCK_SSD 2026-02-25T08:40:48.425 INFO:teuthology.orchestra.run.trial183.stdout: seastore max concurrent transactions = 8 2026-02-25T08:40:48.425 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.425 INFO:teuthology.orchestra.run.trial183.stdout:[mgr] 2026-02-25T08:40:48.426 INFO:teuthology.orchestra.run.trial183.stdout: debug ms = 1 2026-02-25T08:40:48.426 INFO:teuthology.orchestra.run.trial183.stdout: debug mgr = 20 2026-02-25T08:40:48.426 INFO:teuthology.orchestra.run.trial183.stdout: debug mon = 20 2026-02-25T08:40:48.426 INFO:teuthology.orchestra.run.trial183.stdout: debug auth = 20 2026-02-25T08:40:48.426 INFO:teuthology.orchestra.run.trial183.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:48.426 INFO:teuthology.orchestra.run.trial183.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:48.426 INFO:teuthology.orchestra.run.trial183.stdout: mgr/telemetry/nag = false 2026-02-25T08:40:48.426 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.426 INFO:teuthology.orchestra.run.trial183.stdout:[mon] 2026-02-25T08:40:48.426 INFO:teuthology.orchestra.run.trial183.stdout: debug ms = 1 2026-02-25T08:40:48.426 INFO:teuthology.orchestra.run.trial183.stdout: debug mon = 20 2026-02-25T08:40:48.426 INFO:teuthology.orchestra.run.trial183.stdout: debug paxos = 20 2026-02-25T08:40:48.426 INFO:teuthology.orchestra.run.trial183.stdout: debug auth = 20 2026-02-25T08:40:48.427 INFO:teuthology.orchestra.run.trial183.stdout: mon data avail warn = 5 2026-02-25T08:40:48.427 INFO:teuthology.orchestra.run.trial183.stdout: mon mgr mkfs grace = 240 2026-02-25T08:40:48.427 INFO:teuthology.orchestra.run.trial183.stdout: mon reweight min pgs per osd = 4 2026-02-25T08:40:48.427 INFO:teuthology.orchestra.run.trial183.stdout: mon osd reporter subtree level = osd 2026-02-25T08:40:48.427 INFO:teuthology.orchestra.run.trial183.stdout: mon osd prime pg temp = true 2026-02-25T08:40:48.427 INFO:teuthology.orchestra.run.trial183.stdout: mon reweight min bytes per osd = 10 2026-02-25T08:40:48.427 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.427 INFO:teuthology.orchestra.run.trial183.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-25T08:40:48.427 INFO:teuthology.orchestra.run.trial183.stdout: auth mon ticket ttl = 660 # 11m 2026-02-25T08:40:48.427 INFO:teuthology.orchestra.run.trial183.stdout: auth service ticket ttl = 240 # 4m 2026-02-25T08:40:48.427 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.427 INFO:teuthology.orchestra.run.trial183.stdout: # don't complain about insecure global_id in the test suite 2026-02-25T08:40:48.428 INFO:teuthology.orchestra.run.trial183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-25T08:40:48.428 INFO:teuthology.orchestra.run.trial183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-25T08:40:48.428 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.428 INFO:teuthology.orchestra.run.trial183.stdout: # 1m isn't quite enough 2026-02-25T08:40:48.428 INFO:teuthology.orchestra.run.trial183.stdout: mon_down_mkfs_grace = 2m 2026-02-25T08:40:48.428 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.428 INFO:teuthology.orchestra.run.trial183.stdout: mon_warn_on_filestore_osds = false 2026-02-25T08:40:48.428 INFO:teuthology.orchestra.run.trial183.stdout: mon min osdmap epochs = 50 2026-02-25T08:40:48.428 INFO:teuthology.orchestra.run.trial183.stdout: mon osdmap full prune interval = 2 2026-02-25T08:40:48.428 INFO:teuthology.orchestra.run.trial183.stdout: mon osdmap full prune min = 15 2026-02-25T08:40:48.428 INFO:teuthology.orchestra.run.trial183.stdout: mon osdmap full prune txsize = 2 2026-02-25T08:40:48.428 INFO:teuthology.orchestra.run.trial183.stdout: osd pool default crimson = True 2026-02-25T08:40:48.428 INFO:teuthology.orchestra.run.trial183.stdout: paxos service trim min = 10 2026-02-25T08:40:48.429 INFO:teuthology.orchestra.run.trial183.stdout: 2026-02-25T08:40:48.429 INFO:teuthology.orchestra.run.trial183.stdout:[client] 2026-02-25T08:40:48.429 INFO:teuthology.orchestra.run.trial183.stdout: rgw cache enabled = true 2026-02-25T08:40:48.429 INFO:teuthology.orchestra.run.trial183.stdout: rgw enable ops log = true 2026-02-25T08:40:48.429 INFO:teuthology.orchestra.run.trial183.stdout: rgw enable usage log = true 2026-02-25T08:40:48.429 INFO:teuthology.orchestra.run.trial183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-25T08:40:48.429 INFO:teuthology.orchestra.run.trial183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-25T08:40:48.429 INFO:teuthology.orchestra.run.trial183.stdout:[mon.a] 2026-02-25T08:40:48.429 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-25T08:40:48.430 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-25T08:40:48.506 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-25T08:40:48.506 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:40:48.507 DEBUG:teuthology.orchestra.run.trial169:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-25T08:40:48.523 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:40:48.523 DEBUG:teuthology.orchestra.run.trial169:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-25T08:40:48.580 INFO:tasks.ceph:Sending monmap to node ubuntu@trial169.front.sepia.ceph.com 2026-02-25T08:40:48.580 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:40:48.580 DEBUG:teuthology.orchestra.run.trial169:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-25T08:40:48.581 DEBUG:teuthology.orchestra.run.trial169:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:40:48.653 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:40:48.653 DEBUG:teuthology.orchestra.run.trial169:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:48.709 INFO:tasks.ceph:Sending monmap to node ubuntu@trial183.front.sepia.ceph.com 2026-02-25T08:40:48.709 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:40:48.710 DEBUG:teuthology.orchestra.run.trial183:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-25T08:40:48.710 DEBUG:teuthology.orchestra.run.trial183:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-25T08:40:48.744 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:40:48.744 DEBUG:teuthology.orchestra.run.trial183:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:48.799 INFO:tasks.ceph:Setting up mon nodes... 2026-02-25T08:40:48.799 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-25T08:40:48.799 DEBUG:teuthology.orchestra.run.trial183:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T08:40:48.891 INFO:teuthology.orchestra.run.trial183.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-25T08:40:48.894 INFO:tasks.ceph:Setting up mds nodes... 2026-02-25T08:40:48.895 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-25T08:40:48.895 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2026-02-25T08:40:48.938 INFO:teuthology.orchestra.run.trial169.stdout:creating /etc/ceph/ceph.client.0.keyring 2026-02-25T08:40:48.950 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2026-02-25T08:40:48.993 INFO:teuthology.orchestra.run.trial183.stdout:creating /etc/ceph/ceph.client.1.keyring 2026-02-25T08:40:49.005 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-25T08:40:49.005 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial169.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@trial183.front.sepia.ceph.com'): {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'}} 2026-02-25T08:40:49.006 DEBUG:teuthology.orchestra.run.trial169:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:49.029 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:49.029 INFO:tasks.ceph:role: osd.0 2026-02-25T08:40:49.029 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial169.front.sepia.ceph.com 2026-02-25T08:40:49.029 DEBUG:teuthology.orchestra.run.trial169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-25T08:40:49.094 INFO:teuthology.orchestra.run.trial169.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:49.094 INFO:teuthology.orchestra.run.trial169.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:49.094 INFO:teuthology.orchestra.run.trial169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:49.094 INFO:teuthology.orchestra.run.trial169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:49.094 INFO:teuthology.orchestra.run.trial169.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:49.094 INFO:teuthology.orchestra.run.trial169.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:49.094 INFO:teuthology.orchestra.run.trial169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:49.094 INFO:teuthology.orchestra.run.trial169.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:49.094 INFO:teuthology.orchestra.run.trial169.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:49.094 INFO:teuthology.orchestra.run.trial169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:49.664 INFO:teuthology.orchestra.run.trial169.stdout:Discarding blocks...Done. 2026-02-25T08:40:49.666 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial169.front.sepia.ceph.com -o noatime 2026-02-25T08:40:49.666 DEBUG:teuthology.orchestra.run.trial169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:49.738 DEBUG:teuthology.orchestra.run.trial169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:49.804 DEBUG:teuthology.orchestra.run.trial169:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:49.868 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2'} 2026-02-25T08:40:49.868 INFO:tasks.ceph:role: osd.1 2026-02-25T08:40:49.868 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial169.front.sepia.ceph.com 2026-02-25T08:40:49.868 DEBUG:teuthology.orchestra.run.trial169:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-25T08:40:49.932 INFO:teuthology.orchestra.run.trial169.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:40:49.932 INFO:teuthology.orchestra.run.trial169.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:40:49.932 INFO:teuthology.orchestra.run.trial169.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:40:49.932 INFO:teuthology.orchestra.run.trial169.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:40:49.932 INFO:teuthology.orchestra.run.trial169.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:40:49.932 INFO:teuthology.orchestra.run.trial169.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:40:49.932 INFO:teuthology.orchestra.run.trial169.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:40:49.932 INFO:teuthology.orchestra.run.trial169.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:40:49.932 INFO:teuthology.orchestra.run.trial169.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:40:49.932 INFO:teuthology.orchestra.run.trial169.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:40:50.491 INFO:teuthology.orchestra.run.trial169.stdout:Discarding blocks...Done. 2026-02-25T08:40:50.493 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial169.front.sepia.ceph.com -o noatime 2026-02-25T08:40:50.493 DEBUG:teuthology.orchestra.run.trial169:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:50.561 DEBUG:teuthology.orchestra.run.trial169:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-25T08:40:50.625 DEBUG:teuthology.orchestra.run.trial169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:50.777 INFO:teuthology.orchestra.run.trial169.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:40:50.834 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:50,833 seastar - Reactor backend: linux-aio 2026-02-25T08:40:50.850 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:50,849 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:50.850 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:50,850 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:50.852 INFO:teuthology.orchestra.run.trial169.stderr:==39251==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:50.909 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:50,908 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:40:50.909 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:50,908 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:40:50.909 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:50,908 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:40:50.909 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:50,908 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:40:50.909 INFO:teuthology.orchestra.run.trial169.stderr:DEBUG 2026-02-25 08:40:50,909 [shard 0:main] seastar - reactor::drain 2026-02-25T08:40:52.996 INFO:teuthology.orchestra.run.trial169.stderr:----------------------------------------------------- 2026-02-25T08:40:52.996 INFO:teuthology.orchestra.run.trial169.stderr:Suppressions used: 2026-02-25T08:40:52.997 INFO:teuthology.orchestra.run.trial169.stderr: count bytes template 2026-02-25T08:40:52.997 INFO:teuthology.orchestra.run.trial169.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:52.997 INFO:teuthology.orchestra.run.trial169.stderr:----------------------------------------------------- 2026-02-25T08:40:52.997 INFO:teuthology.orchestra.run.trial169.stderr: 2026-02-25T08:40:53.016 INFO:teuthology.orchestra.run.trial169.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:40:53.016 INFO:teuthology.orchestra.run.trial169.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:40:53.054 INFO:teuthology.orchestra.run.trial169.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:40:53.054 INFO:teuthology.orchestra.run.trial169.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:40:53.061 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:53,060 seastar - Reactor backend: linux-aio 2026-02-25T08:40:53.074 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:53,073 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:53.074 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:53,073 [shard 3:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:53.078 INFO:teuthology.orchestra.run.trial169.stderr:==39249==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:53.078 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:53,078 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:40:53.141 INFO:teuthology.orchestra.run.trial169.stderr:DEBUG 2026-02-25 08:40:53,141 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:40:53.221 INFO:teuthology.orchestra.run.trial169.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1583b 0x10a2823 0xf47938 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:53.221 INFO:teuthology.orchestra.run.trial169.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9cf61eb 0x9d73986 0x9d73a9c 0x9d73b78 0x9d74234 0x9d73ead 0x9d7439b 0x9d7479f 0x9d74a5c 0x9f165e1 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:53.221 INFO:teuthology.orchestra.run.trial169.stderr:kernel callstack: 0xffffffffffffff80 0xffffffff8a29431b 0xffffffff8a420057 0xffffffff8a4203b0 0xffffffff8a29525e 0xffffffff8a5e7269 0xffffffff8a5ca7d5 0xffffffff8a5d4bab 0xffffffff8a5d500b 0xffffffff8a2910d2 0xffffffff8af50451 0xffffffff8b000ba2 2026-02-25T08:40:53.221 INFO:teuthology.orchestra.run.trial169.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x10a26f2 0xf47938 0x9e218ac 0x9e21ba9 0x9e21e1c 0x9e23b43 0x9e24090 0x9e2428f 0x9f095b1 0x9f09d0c 0x9f13f9a 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:53.380 INFO:teuthology.orchestra.run.trial169.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xb051e /lib64/libasan.so.8+0xb1437 0xb7ac87 0x10a27ac 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:53.380 INFO:teuthology.orchestra.run.trial169.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1190fb /lib64/libasan.so.8+0xc5f4 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe2f10b 0xe2f42e 0x9ef6eb8 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:53.380 INFO:teuthology.orchestra.run.trial169.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15197 0x11380c8 0xf4792d 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e22e1c 0x9e2338a 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:55.531 INFO:teuthology.orchestra.run.trial169.stderr:----------------------------------------------------- 2026-02-25T08:40:55.531 INFO:teuthology.orchestra.run.trial169.stderr:Suppressions used: 2026-02-25T08:40:55.531 INFO:teuthology.orchestra.run.trial169.stderr: count bytes template 2026-02-25T08:40:55.531 INFO:teuthology.orchestra.run.trial169.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:55.531 INFO:teuthology.orchestra.run.trial169.stderr:----------------------------------------------------- 2026-02-25T08:40:55.532 INFO:teuthology.orchestra.run.trial169.stderr: 2026-02-25T08:40:55.561 DEBUG:teuthology.orchestra.run.trial169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-25T08:40:55.588 DEBUG:teuthology.orchestra.run.trial169:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:40:55.739 INFO:teuthology.orchestra.run.trial169.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:40:55.800 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:55,800 seastar - Reactor backend: linux-aio 2026-02-25T08:40:55.810 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:55,809 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:55.810 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:55,809 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:55.812 INFO:teuthology.orchestra.run.trial169.stderr:==39320==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:55.868 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:55,867 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:40:55.868 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:55,867 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:40:55.868 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:55,867 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:40:55.868 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:55,867 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:40:55.868 INFO:teuthology.orchestra.run.trial169.stderr:DEBUG 2026-02-25 08:40:55,867 [shard 0:main] seastar - reactor::drain 2026-02-25T08:40:57.963 INFO:teuthology.orchestra.run.trial169.stderr:----------------------------------------------------- 2026-02-25T08:40:57.963 INFO:teuthology.orchestra.run.trial169.stderr:Suppressions used: 2026-02-25T08:40:57.963 INFO:teuthology.orchestra.run.trial169.stderr: count bytes template 2026-02-25T08:40:57.963 INFO:teuthology.orchestra.run.trial169.stderr: 1 8 MallocExtension::Register 2026-02-25T08:40:57.963 INFO:teuthology.orchestra.run.trial169.stderr:----------------------------------------------------- 2026-02-25T08:40:57.963 INFO:teuthology.orchestra.run.trial169.stderr: 2026-02-25T08:40:57.982 INFO:teuthology.orchestra.run.trial169.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:40:57.982 INFO:teuthology.orchestra.run.trial169.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:40:58.030 INFO:teuthology.orchestra.run.trial169.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:40:58.030 INFO:teuthology.orchestra.run.trial169.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:40:58.037 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:58,036 seastar - Reactor backend: linux-aio 2026-02-25T08:40:58.051 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:58,050 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:58.051 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:58,051 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:40:58.055 INFO:teuthology.orchestra.run.trial169.stderr:==39318==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:40:58.055 INFO:teuthology.orchestra.run.trial169.stderr:INFO 2026-02-25 08:40:58,054 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:40:58.119 INFO:teuthology.orchestra.run.trial169.stderr:DEBUG 2026-02-25 08:40:58,118 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:40:58.200 INFO:teuthology.orchestra.run.trial169.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xea747 0xe40c90 0xe4104b 0x9f19999 0x78b69db 0x7ba11e3 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:58.200 INFO:teuthology.orchestra.run.trial169.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9d73989 0x9d73a9c 0x9d73b78 0x9d74234 0x9d73ead 0x9d7439b 0x9d7479f 0x9d74a5c 0x9f165e1 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:58.200 INFO:teuthology.orchestra.run.trial169.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x10a2720 0xf47938 0x9e216df 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22694 0x9e2ab3b 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:58.359 INFO:teuthology.orchestra.run.trial169.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9e21643 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:58.359 INFO:teuthology.orchestra.run.trial169.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xe29e5c 0x1138071 0xf4792d 0x9f044e9 0x9f04ad6 0x9f04e59 0x9f14fb0 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:40:58.359 INFO:teuthology.orchestra.run.trial169.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9e21dba 0x9e220d2 0x9e22694 0x9e2ab3b 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:00.506 INFO:teuthology.orchestra.run.trial169.stderr:----------------------------------------------------- 2026-02-25T08:41:00.506 INFO:teuthology.orchestra.run.trial169.stderr:Suppressions used: 2026-02-25T08:41:00.507 INFO:teuthology.orchestra.run.trial169.stderr: count bytes template 2026-02-25T08:41:00.507 INFO:teuthology.orchestra.run.trial169.stderr: 1 8 MallocExtension::Register 2026-02-25T08:41:00.507 INFO:teuthology.orchestra.run.trial169.stderr:----------------------------------------------------- 2026-02-25T08:41:00.507 INFO:teuthology.orchestra.run.trial169.stderr: 2026-02-25T08:41:00.538 DEBUG:teuthology.orchestra.run.trial169:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-25T08:41:00.564 DEBUG:teuthology.orchestra.run.trial183:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-25T08:41:00.590 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-25T08:41:00.590 INFO:tasks.ceph:role: osd.2 2026-02-25T08:41:00.591 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial183.front.sepia.ceph.com 2026-02-25T08:41:00.591 DEBUG:teuthology.orchestra.run.trial183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-25T08:41:00.656 INFO:teuthology.orchestra.run.trial183.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:41:00.657 INFO:teuthology.orchestra.run.trial183.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:41:00.657 INFO:teuthology.orchestra.run.trial183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:41:00.657 INFO:teuthology.orchestra.run.trial183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:41:00.657 INFO:teuthology.orchestra.run.trial183.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:41:00.657 INFO:teuthology.orchestra.run.trial183.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:41:00.657 INFO:teuthology.orchestra.run.trial183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:41:00.658 INFO:teuthology.orchestra.run.trial183.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:41:00.658 INFO:teuthology.orchestra.run.trial183.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:41:00.658 INFO:teuthology.orchestra.run.trial183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:41:01.235 INFO:teuthology.orchestra.run.trial183.stdout:Discarding blocks...Done. 2026-02-25T08:41:01.237 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial183.front.sepia.ceph.com -o noatime 2026-02-25T08:41:01.237 DEBUG:teuthology.orchestra.run.trial183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-2 2026-02-25T08:41:01.306 DEBUG:teuthology.orchestra.run.trial183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-25T08:41:01.374 DEBUG:teuthology.orchestra.run.trial183:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-02-25T08:41:01.439 INFO:tasks.ceph:roles_to_devs: {'osd.2': '/dev/vg_nvme/lv_1', 'osd.3': '/dev/vg_nvme/lv_2'} 2026-02-25T08:41:01.439 INFO:tasks.ceph:role: osd.3 2026-02-25T08:41:01.440 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial183.front.sepia.ceph.com 2026-02-25T08:41:01.440 DEBUG:teuthology.orchestra.run.trial183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-25T08:41:01.504 INFO:teuthology.orchestra.run.trial183.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-25T08:41:01.504 INFO:teuthology.orchestra.run.trial183.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-25T08:41:01.504 INFO:teuthology.orchestra.run.trial183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-25T08:41:01.504 INFO:teuthology.orchestra.run.trial183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2026-02-25T08:41:01.504 INFO:teuthology.orchestra.run.trial183.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-25T08:41:01.504 INFO:teuthology.orchestra.run.trial183.stdout: = sunit=0 swidth=0 blks 2026-02-25T08:41:01.504 INFO:teuthology.orchestra.run.trial183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-25T08:41:01.505 INFO:teuthology.orchestra.run.trial183.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-25T08:41:01.505 INFO:teuthology.orchestra.run.trial183.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-25T08:41:01.505 INFO:teuthology.orchestra.run.trial183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-25T08:41:02.063 INFO:teuthology.orchestra.run.trial183.stdout:Discarding blocks...Done. 2026-02-25T08:41:02.065 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial183.front.sepia.ceph.com -o noatime 2026-02-25T08:41:02.066 DEBUG:teuthology.orchestra.run.trial183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-3 2026-02-25T08:41:02.134 DEBUG:teuthology.orchestra.run.trial183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-02-25T08:41:02.203 DEBUG:teuthology.orchestra.run.trial183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:41:02.355 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:41:02.419 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:02,418 seastar - Reactor backend: linux-aio 2026-02-25T08:41:02.437 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:02,436 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:02.437 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:02,436 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:02.439 INFO:teuthology.orchestra.run.trial183.stderr:==39141==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:41:02.496 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:02,495 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:41:02.496 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:02,495 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:41:02.496 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:02,495 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:41:02.496 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:02,495 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:41:02.497 INFO:teuthology.orchestra.run.trial183.stderr:DEBUG 2026-02-25 08:41:02,495 [shard 0:main] seastar - reactor::drain 2026-02-25T08:41:04.552 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-25T08:41:04.553 INFO:teuthology.orchestra.run.trial183.stderr:Suppressions used: 2026-02-25T08:41:04.553 INFO:teuthology.orchestra.run.trial183.stderr: count bytes template 2026-02-25T08:41:04.553 INFO:teuthology.orchestra.run.trial183.stderr: 1 8 MallocExtension::Register 2026-02-25T08:41:04.553 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-25T08:41:04.553 INFO:teuthology.orchestra.run.trial183.stderr: 2026-02-25T08:41:04.572 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:41:04.572 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:41:04.619 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:41:04.619 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:41:04.626 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:04,625 seastar - Reactor backend: linux-aio 2026-02-25T08:41:04.634 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:04,634 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:04.634 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:04,634 [shard 1:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:04.639 INFO:teuthology.orchestra.run.trial183.stderr:==39139==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:41:04.639 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:04,638 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:41:04.697 INFO:teuthology.orchestra.run.trial183.stderr:DEBUG 2026-02-25 08:41:04,696 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:41:04.776 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x3617f04 0x9e33da7 0x9e33e69 0x9ef3c57 0x9ef3dec 0x9ef417e 0x9ef5765 0x9ef6a80 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:04.776 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc65c2 /lib64/libasan.so.8+0x157c7 0x10a2823 0xf47938 0x9e253ca 0x9e25d09 0x9f0fc51 0x9f1a337 0x78b69db 0x7ba17e8 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:04.776 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc597 /lib64/libasan.so.8+0x91c1 /lib64/libasan.so.8+0xee7dc 0x9cf61eb 0x9d73986 0x9d73a9c 0x9d73b78 0x9d73df4 0x9d73ead 0x9d7439b 0x9d7479f 0x9d74a5c 0x9f165e1 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:04.776 INFO:teuthology.orchestra.run.trial183.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa4291052 0xffffffffa4f50451 0xffffffffa5000ba2 2026-02-25T08:41:04.927 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x119134 /lib64/libasan.so.8+0xebbf /lib64/libasan.so.8+0xef905 0x7453a06 0x711b579 0x711bb38 0x78b6a18 0x7ba11e3 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:04.927 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x164fb 0x9f165f0 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:04.928 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x119826 /lib64/libasan.so.8+0xfd239 /lib64/libasan.so.8+0xef979 0xbc88cc 0xbc8946 0xbc8a5f 0x9ef6ec1 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:07.111 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-25T08:41:07.112 INFO:teuthology.orchestra.run.trial183.stderr:Suppressions used: 2026-02-25T08:41:07.112 INFO:teuthology.orchestra.run.trial183.stderr: count bytes template 2026-02-25T08:41:07.112 INFO:teuthology.orchestra.run.trial183.stderr: 1 8 MallocExtension::Register 2026-02-25T08:41:07.112 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-25T08:41:07.112 INFO:teuthology.orchestra.run.trial183.stderr: 2026-02-25T08:41:07.143 DEBUG:teuthology.orchestra.run.trial183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-25T08:41:07.168 DEBUG:teuthology.orchestra.run.trial183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:41:07.321 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: parsing early config 2026-02-25T08:41:07.374 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:07,373 seastar - Reactor backend: linux-aio 2026-02-25T08:41:07.392 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:07,391 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:07.392 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:07,391 [shard 0:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:07.394 INFO:teuthology.orchestra.run.trial183.stderr:==39208==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:41:07.450 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:07,448 [shard 0:main] osd - bypassing the config fetch due to --no-mon-config 2026-02-25T08:41:07.450 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:07,448 [shard 0:main] osd - Configure option_name crimson_reactor_task_quota_ms with value : 0.500000 2026-02-25T08:41:07.450 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:07,448 [shard 0:main] osd - Configure option_name crimson_reactor_idle_poll_time_us with value : 200 2026-02-25T08:41:07.450 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:07,448 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 4 2026-02-25T08:41:07.450 INFO:teuthology.orchestra.run.trial183.stderr:DEBUG 2026-02-25 08:41:07,449 [shard 0:main] seastar - reactor::drain 2026-02-25T08:41:09.491 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-25T08:41:09.491 INFO:teuthology.orchestra.run.trial183.stderr:Suppressions used: 2026-02-25T08:41:09.491 INFO:teuthology.orchestra.run.trial183.stderr: count bytes template 2026-02-25T08:41:09.491 INFO:teuthology.orchestra.run.trial183.stderr: 1 8 MallocExtension::Register 2026-02-25T08:41:09.491 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-25T08:41:09.491 INFO:teuthology.orchestra.run.trial183.stderr: 2026-02-25T08:41:09.510 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: early config parsed successfully 2026-02-25T08:41:09.510 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: initializing seastar app_template 2026-02-25T08:41:09.552 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: registering CLI options 2026-02-25T08:41:09.552 INFO:teuthology.orchestra.run.trial183.stderr:INFO  osd - OSD::main: entering seastar runtime 2026-02-25T08:41:09.559 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:09,559 seastar - Reactor backend: linux-aio 2026-02-25T08:41:09.569 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:09,569 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:09.569 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:09,569 [shard 2:main] io - IO queue was unable to find a suitable maximum request length, the search was cut-off early at: 16MB 2026-02-25T08:41:09.573 INFO:teuthology.orchestra.run.trial183.stderr:==39206==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases! 2026-02-25T08:41:09.573 INFO:teuthology.orchestra.run.trial183.stderr:INFO 2026-02-25 08:41:09,573 [shard 0:main] osd - OSD::main: seastar runtime started 2026-02-25T08:41:09.633 INFO:teuthology.orchestra.run.trial183.stderr:DEBUG 2026-02-25 08:41:09,632 [shard 0:main] osd - OSD::main: initializing logger output 2026-02-25T08:41:09.712 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x361a4d7 0x361a859 0x9ef6a64 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:09.712 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0xc589 /lib64/libasan.so.8+0x8eea /lib64/libasan.so.8+0xedc45 0xe40f5d 0x9f16c08 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:09.712 INFO:teuthology.orchestra.run.trial183.stderr:kernel callstack: 0xffffffffffffff80 0xffffffffa4291052 0xffffffffa4f50451 0xffffffffa5000ba2 2026-02-25T08:41:09.712 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x35fc14c 0x35fc215 0x3618092 0x9e33da7 0x9e33eaf 0x9ef3c57 0x9ef3dec 0x9ef417e 0x9ef5765 0x9ef6a80 0x9ef6ea0 0x9f1372b 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7044244 0x7048045 0x704845b 0x73b7c8f 0x73b7d14 0x73b7e16 0x73b7f0e 0x74780b4 0x72214da 0x7266e18 0x7398e9a 0x73efcd2 0x7221561 0x726701f 0x755dcc5 0x73b7f93 0x73b8071 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:09.753 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0xcc2652 0x73c0b1a 0x73e68b0 0x711bb0c 0x7ba1b76 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x7045030 0x74fc8bc 0x74fca10 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:09.865 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 2, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libc.so.6+0x8cc08 /lib64/libasan.so.8+0x10069c /lib64/libasan.so.8+0xc65b9 /lib64/libasan.so.8+0x15197 0x9e21748 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22694 0x9e2ab3b 0x9e2b21b 0x9f13d68 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:09.865 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 3, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f /lib64/libasan.so.8+0x1297e4 /lib64/libasan.so.8+0xc65d1 /lib64/libasan.so.8+0x15e37 0x9f04c51 0x9f04e59 0x9f14fb0 0x9f18413 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:09.865 INFO:teuthology.orchestra.run.trial183.stderr:Reactor stalled for 32 ms on shard 1, in scheduling group main. Backtrace: /lib64/libasan.so.8+0x54e93 0x9ff8dc8 0x9ff90b1 0xa059a4f 0xa05a242 0xa05a4b0 0xa05a5f8 0xa05aab7 /lib64/libc.so.6+0x3fc2f 0x9e21977 0x9e21ba9 0x9e21e1c 0x9e220d2 0x9e22b81 0x9e2307f 0x9e236af 0x9e23819 0x9f03b89 0x9f1698c 0x9ed89e1 0x9ed79e1 0x7ba18b1 0x7bab3d4 0x7a28841 0x7a561e3 0x716287f 0x6fec95e 0x6ed2592 0x73b5492 0x73b5517 0x73b5619 0x73b5711 0x7477f38 0x72201a5 0x7260160 0x73973b5 0x73ea713 0x722022c 0x7260367 0x755b9c5 0x73b5796 0x73b5874 0x7397530 0x7495d5a 0x7529f3a 0x7529fb5 0x755ba46 0x747862e 0x75ba6f2 0xa0b3917 0xa007bec 0xa0e1387 0xa149b6a 0xa1645fa 0xa164c76 0x272f695 0xa772b80 /lib64/libasan.so.8+0x2a549 /lib64/libc.so.6+0x8b2f9 /lib64/libc.so.6+0x1103ff 2026-02-25T08:41:12.016 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-25T08:41:12.016 INFO:teuthology.orchestra.run.trial183.stderr:Suppressions used: 2026-02-25T08:41:12.016 INFO:teuthology.orchestra.run.trial183.stderr: count bytes template 2026-02-25T08:41:12.017 INFO:teuthology.orchestra.run.trial183.stderr: 1 8 MallocExtension::Register 2026-02-25T08:41:12.017 INFO:teuthology.orchestra.run.trial183.stderr:----------------------------------------------------- 2026-02-25T08:41:12.017 INFO:teuthology.orchestra.run.trial183.stderr: 2026-02-25T08:41:12.047 DEBUG:teuthology.orchestra.run.trial183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-25T08:41:12.074 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-25T08:41:12.074 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:41:12.074 DEBUG:teuthology.orchestra.run.trial169:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-25T08:41:12.102 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:41:12.102 DEBUG:teuthology.orchestra.run.trial169:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-25T08:41:12.166 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:41:12.166 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-25T08:41:12.193 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:41:12.193 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-25T08:41:12.257 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:41:12.257 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-02-25T08:41:12.321 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:41:12.321 DEBUG:teuthology.orchestra.run.trial169:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2026-02-25T08:41:12.339 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:41:12.339 DEBUG:teuthology.orchestra.run.trial183:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2026-02-25T08:41:12.377 INFO:tasks.ceph:Adding keys to all mons... 2026-02-25T08:41:12.378 DEBUG:teuthology.orchestra.run.trial169:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-25T08:41:12.402 INFO:teuthology.orchestra.run.trial169.stdout:[osd.0] 2026-02-25T08:41:12.402 INFO:teuthology.orchestra.run.trial169.stdout: key = AQCVtZ5pxTWkCBAAZsV9AmrdIU5+2Yi3GxyzSw== 2026-02-25T08:41:12.402 INFO:teuthology.orchestra.run.trial169.stdout:[osd.1] 2026-02-25T08:41:12.402 INFO:teuthology.orchestra.run.trial169.stdout: key = AQCatZ5p+KFSBxAA8ZkFMiExPh4ECib+E5/SlQ== 2026-02-25T08:41:12.402 INFO:teuthology.orchestra.run.trial169.stdout:[mgr.x] 2026-02-25T08:41:12.402 INFO:teuthology.orchestra.run.trial169.stdout: key = AQCQtZ5p2cMYNRAAJqs9Idt+MYFFAIE1xSCeNA== 2026-02-25T08:41:12.403 INFO:teuthology.orchestra.run.trial169.stdout:[osd.2] 2026-02-25T08:41:12.403 INFO:teuthology.orchestra.run.trial169.stdout: key = AQCgtZ5p8K64KRAAIb0hWaFegnMio7TkyK+FhA== 2026-02-25T08:41:12.403 INFO:teuthology.orchestra.run.trial169.stdout:[osd.3] 2026-02-25T08:41:12.403 INFO:teuthology.orchestra.run.trial169.stdout: key = AQCltZ5pcPjtJRAAF+f5gr7+Znub1HG0patJtA== 2026-02-25T08:41:12.403 INFO:teuthology.orchestra.run.trial169.stdout:[client.0] 2026-02-25T08:41:12.403 INFO:teuthology.orchestra.run.trial169.stdout: key = AQCQtZ5psmLtNxAAnh3CAjfwGrq8XovSch7xWQ== 2026-02-25T08:41:12.403 INFO:teuthology.orchestra.run.trial169.stdout:[client.1] 2026-02-25T08:41:12.403 INFO:teuthology.orchestra.run.trial169.stdout: key = AQCQtZ5peogvOxAAFa3Nw5vfEvrpH7lhbDZ3Ew== 2026-02-25T08:41:12.404 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:41:12.494 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:41:12.541 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-25T08:41:12.589 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:41:12.637 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-25T08:41:12.685 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-25T08:41:12.735 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-25T08:41:12.786 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-25T08:41:12.787 DEBUG:teuthology.orchestra.run.trial169:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-25T08:41:12.812 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-25T08:41:12.899 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:12.897+0000 7f4c2aea9c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:12.899 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:12.897+0000 7f4c2aea9c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:12.911 DEBUG:teuthology.orchestra.run.trial169:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-25T08:41:12.936 DEBUG:teuthology.orchestra.run.trial169:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-25T08:41:12.993 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-25T08:41:12.993 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-25T08:41:12.993 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-25T08:41:13.035 INFO:tasks.ceph.mon.a:Started 2026-02-25T08:41:13.035 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-25T08:41:13.035 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-25T08:41:13.035 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-25T08:41:13.038 INFO:tasks.ceph.mgr.x:Started 2026-02-25T08:41:13.038 DEBUG:tasks.ceph:set 0 configs 2026-02-25T08:41:13.039 DEBUG:teuthology.orchestra.run.trial169:> sudo ceph --cluster ceph config dump 2026-02-25T08:41:13.098 INFO:tasks.ceph.mon.a.trial169.stderr:2026-02-25T08:41:13.095+0000 7f3f04087c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.098 INFO:tasks.ceph.mon.a.trial169.stderr:2026-02-25T08:41:13.096+0000 7f3f04087c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.098 INFO:tasks.ceph.mon.a.trial169.stderr:2026-02-25T08:41:13.096+0000 7f3f04087c00 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.099 INFO:tasks.ceph.mgr.x.trial183.stderr:2026-02-25T08:41:13.097+0000 7f3049835fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.113 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:13.108+0000 7f16fbe11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.116 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:13.115+0000 7f16fbe11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.122 INFO:tasks.ceph.mgr.x.trial183.stderr:2026-02-25T08:41:13.120+0000 7f3049835fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.123 INFO:tasks.ceph.mgr.x.trial183.stderr:2026-02-25T08:41:13.121+0000 7f3049835fc0 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.149 INFO:teuthology.orchestra.run.trial169.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-25T08:41:13.158 INFO:tasks.ceph:Setting crush tunables to default 2026-02-25T08:41:13.158 DEBUG:teuthology.orchestra.run.trial169:> sudo ceph --cluster ceph osd crush tunables default 2026-02-25T08:41:13.233 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:13.228+0000 7fba09e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.235 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:13.234+0000 7fba09e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.270 INFO:teuthology.orchestra.run.trial169.stderr:adjusted tunables profile to default 2026-02-25T08:41:13.278 INFO:tasks.ceph:check_enable_crimson: True 2026-02-25T08:41:13.278 INFO:tasks.ceph:check_enable_crimson: setting set-allow-crimson 2026-02-25T08:41:13.279 DEBUG:teuthology.orchestra.run.trial169:> sudo ceph --cluster ceph osd set-allow-crimson --yes-i-really-mean-it 2026-02-25T08:41:13.353 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:13.349+0000 7f40b5f8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.356 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:13.355+0000 7f40b5f8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:13.396 INFO:tasks.ceph.mon.a.trial169.stderr:2026-02-25T08:41:13.395+0000 7f3efb7f1640 -1 WARNING: experimental feature 'crimson' is enabled 2026-02-25T08:41:13.396 INFO:tasks.ceph.mon.a.trial169.stderr:Please be aware that this feature is experimental, untested, 2026-02-25T08:41:13.397 INFO:tasks.ceph.mon.a.trial169.stderr:unsupported, and may result in data corruption, data loss, 2026-02-25T08:41:13.397 INFO:tasks.ceph.mon.a.trial169.stderr:and/or irreparable damage to your cluster. Do not use 2026-02-25T08:41:13.397 INFO:tasks.ceph.mon.a.trial169.stderr:feature with important data. 2026-02-25T08:41:13.397 INFO:tasks.ceph.mon.a.trial169.stderr: 2026-02-25T08:41:13.962 INFO:tasks.ceph.mgr.x.trial183.stderr:/usr/lib64/python3.9/site-packages/scipy/__init__.py:73: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-25T08:41:13.962 INFO:tasks.ceph.mgr.x.trial183.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-25T08:41:13.962 INFO:tasks.ceph.mgr.x.trial183.stderr: from numpy import show_config as show_numpy_config 2026-02-25T08:41:14.282 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-25T08:41:14.283 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:41:14.283 DEBUG:teuthology.orchestra.run.trial169:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-25T08:41:14.308 DEBUG:teuthology.orchestra.run.trial169:> set -ex 2026-02-25T08:41:14.308 DEBUG:teuthology.orchestra.run.trial169:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-25T08:41:14.371 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:41:14.371 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-25T08:41:14.397 DEBUG:teuthology.orchestra.run.trial183:> set -ex 2026-02-25T08:41:14.397 DEBUG:teuthology.orchestra.run.trial183:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-02-25T08:41:14.461 DEBUG:teuthology.orchestra.run.trial183:> sudo ceph --cluster ceph osd new c2af3fce-0bbe-497c-8eb5-86ae390e5605 0 2026-02-25T08:41:14.575 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-25T08:41:14.570+0000 7f48da9ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:14.577 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-25T08:41:14.575+0000 7f48da9ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:14.615 INFO:teuthology.orchestra.run.trial183.stdout:0 2026-02-25T08:41:14.624 DEBUG:teuthology.orchestra.run.trial183:> sudo ceph --cluster ceph osd new 1c19cfc2-9b80-4f78-88a9-7d7477f24d4f 1 2026-02-25T08:41:14.698 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-25T08:41:14.693+0000 7fcb635c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:14.700 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-25T08:41:14.698+0000 7fcb635c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:14.738 INFO:teuthology.orchestra.run.trial183.stdout:1 2026-02-25T08:41:14.747 DEBUG:teuthology.orchestra.run.trial183:> sudo ceph --cluster ceph osd new 61930931-bbe8-4bf2-9b0a-eebae4148e87 2 2026-02-25T08:41:14.819 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-25T08:41:14.814+0000 7f388edfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:14.822 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-25T08:41:14.820+0000 7f388edfd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:14.859 INFO:teuthology.orchestra.run.trial183.stdout:2 2026-02-25T08:41:14.868 DEBUG:teuthology.orchestra.run.trial183:> sudo ceph --cluster ceph osd new 02f3ca2c-35c9-491e-9310-1a74c4aa8c04 3 2026-02-25T08:41:14.942 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-25T08:41:14.937+0000 7f7bf125a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:14.945 INFO:teuthology.orchestra.run.trial183.stderr:2026-02-25T08:41:14.943+0000 7f7bf125a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:14.984 INFO:teuthology.orchestra.run.trial183.stdout:3 2026-02-25T08:41:14.992 DEBUG:teuthology.orchestra.run.trial169:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.0.log 2026-02-25T08:41:15.023 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-25T08:41:15.024 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-25T08:41:15.066 INFO:tasks.ceph.osd.0:Started 2026-02-25T08:41:15.066 DEBUG:teuthology.orchestra.run.trial169:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.1.log 2026-02-25T08:41:15.094 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-25T08:41:15.094 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2>> /var/log/ceph/ceph-osd.1.log 2026-02-25T08:41:15.137 INFO:tasks.ceph.osd.1:Started 2026-02-25T08:41:15.137 DEBUG:teuthology.orchestra.run.trial183:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.2.log 2026-02-25T08:41:15.166 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-25T08:41:15.167 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log 2026-02-25T08:41:15.208 INFO:tasks.ceph.osd.2:Started 2026-02-25T08:41:15.209 DEBUG:teuthology.orchestra.run.trial183:> sudo install -b -o ubuntu -g ubuntu /dev/null /var/log/ceph/ceph-osd.3.log 2026-02-25T08:41:15.238 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-25T08:41:15.238 DEBUG:teuthology.orchestra.run.trial183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log 2026-02-25T08:41:15.280 INFO:tasks.ceph.osd.3:Started 2026-02-25T08:41:15.281 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:15.356 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:15.352+0000 7f08f570b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:15.359 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:15.357+0000 7f08f570b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:15.392 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:15.393 INFO:teuthology.orchestra.run.trial169.stdout:{"epoch":7,"fsid":"a3da792f-4239-4b1a-ad7d-525e6c05ee6a","created":"2026-02-25T08:41:13.109418+0000","modified":"2026-02-25T08:41:14.983015+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:41:14.983015+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c2af3fce-0bbe-497c-8eb5-86ae390e5605","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c19cfc2-9b80-4f78-88a9-7d7477f24d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"61930931-bbe8-4bf2-9b0a-eebae4148e87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"02f3ca2c-35c9-491e-9310-1a74c4aa8c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:15.401 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-25T08:41:15.401 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-25T08:41:15.805 DEBUG:teuthology.orchestra.run.trial169:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:15.872 INFO:teuthology.misc.health.trial169.stderr:2026-02-25T08:41:15.869+0000 7f556e70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:15.875 INFO:teuthology.misc.health.trial169.stderr:2026-02-25T08:41:15.873+0000 7f556e70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:15.908 INFO:teuthology.misc.health.trial169.stdout: 2026-02-25T08:41:15.908 INFO:teuthology.misc.health.trial169.stdout:{"epoch":7,"fsid":"a3da792f-4239-4b1a-ad7d-525e6c05ee6a","created":"2026-02-25T08:41:13.109418+0000","modified":"2026-02-25T08:41:14.983015+0000","last_up_change":"0.000000","last_in_change":"2026-02-25T08:41:14.983015+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c2af3fce-0bbe-497c-8eb5-86ae390e5605","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1c19cfc2-9b80-4f78-88a9-7d7477f24d4f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"61930931-bbe8-4bf2-9b0a-eebae4148e87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"02f3ca2c-35c9-491e-9310-1a74c4aa8c04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:15.914 DEBUG:teuthology.misc:0 of 4 OSDs are up 2026-02-25T08:41:16.328 INFO:tasks.ceph.mgr.x.trial183.stderr:2026-02-25T08:41:16.326+0000 7f2f6439a640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-25T08:41:22.317 DEBUG:teuthology.orchestra.run.trial169:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:22.390 INFO:teuthology.misc.health.trial169.stderr:2026-02-25T08:41:22.386+0000 7fe1c2f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:22.392 INFO:teuthology.misc.health.trial169.stderr:2026-02-25T08:41:22.390+0000 7fe1c2f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:22.502 INFO:teuthology.misc.health.trial169.stdout: 2026-02-25T08:41:22.503 INFO:teuthology.misc.health.trial169.stdout:{"epoch":11,"fsid":"a3da792f-4239-4b1a-ad7d-525e6c05ee6a","created":"2026-02-25T08:41:13.109418+0000","modified":"2026-02-25T08:41:22.339931+0000","last_up_change":"2026-02-25T08:41:22.339931+0000","last_in_change":"2026-02-25T08:41:14.983015+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[],"osds":[{"osd":0,"uuid":"c2af3fce-0bbe-497c-8eb5-86ae390e5605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6806","nonce":548378417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6805","nonce":548378417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6808","nonce":548378417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6807","nonce":548378417}]},"public_addr":"10.20.193.169:6806/548378417","cluster_addr":"10.20.193.169:6805/548378417","heartbeat_back_addr":"10.20.193.169:6808/548378417","heartbeat_front_addr":"10.20.193.169:6807/548378417","state":["exists","up"]},{"osd":1,"uuid":"1c19cfc2-9b80-4f78-88a9-7d7477f24d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6800","nonce":1807020326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6801","nonce":1807020326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6803","nonce":1807020326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6802","nonce":1807020326}]},"public_addr":"10.20.193.169:6800/1807020326","cluster_addr":"10.20.193.169:6801/1807020326","heartbeat_back_addr":"10.20.193.169:6803/1807020326","heartbeat_front_addr":"10.20.193.169:6802/1807020326","state":["exists","up"]},{"osd":2,"uuid":"61930931-bbe8-4bf2-9b0a-eebae4148e87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6803","nonce":406720561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6802","nonce":406720561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6807","nonce":406720561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6804","nonce":406720561}]},"public_addr":"10.20.193.183:6803/406720561","cluster_addr":"10.20.193.183:6802/406720561","heartbeat_back_addr":"10.20.193.183:6807/406720561","heartbeat_front_addr":"10.20.193.183:6804/406720561","state":["exists","up"]},{"osd":3,"uuid":"02f3ca2c-35c9-491e-9310-1a74c4aa8c04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6805","nonce":1517795374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6806","nonce":1517795374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6809","nonce":1517795374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6808","nonce":1517795374}]},"public_addr":"10.20.193.183:6805/1517795374","cluster_addr":"10.20.193.183:6806/1517795374","heartbeat_back_addr":"10.20.193.183:6809/1517795374","heartbeat_front_addr":"10.20.193.183:6808/1517795374","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:22.510 DEBUG:teuthology.misc:4 of 4 OSDs are up 2026-02-25T08:41:22.510 INFO:tasks.ceph:Creating RBD pool 2026-02-25T08:41:22.511 DEBUG:teuthology.orchestra.run.trial169:> sudo ceph --cluster ceph osd pool create rbd 8 2026-02-25T08:41:22.586 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:22.580+0000 7f68b930b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:22.588 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:22.587+0000 7f68b930b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.351 INFO:teuthology.orchestra.run.trial169.stderr:pool 'rbd' created 2026-02-25T08:41:23.364 DEBUG:teuthology.orchestra.run.trial169:> rbd --cluster ceph pool init rbd 2026-02-25T08:41:23.407 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:23.406+0000 7efe88219240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.408 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:23.406+0000 7efe88219240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:23.411 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:23.409+0000 7efe88219240 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.389 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-25T08:41:26.390 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2026-02-25T08:41:26.391 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-25T08:41:26.467 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:26.463+0000 7f60bf564640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.470 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:26.468+0000 7f60bf564640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.581 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:26.590 INFO:teuthology.orchestra.run.trial169.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-25T08:41:26.591 INFO:tasks.ceph_manager:config epoch is 1 2026-02-25T08:41:26.591 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-25T08:41:26.591 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-25T08:41:26.591 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-25T08:41:26.669 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:26.665+0000 7f8e7210b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.672 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:26.670+0000 7f8e7210b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.815 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:26.823 INFO:teuthology.orchestra.run.trial169.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6800","nonce":366714864},{"type":"v1","addr":"10.20.193.183:6801","nonce":366714864}]},"active_addr":"10.20.193.183:6801/366714864","active_change":"2026-02-25T08:41:15.321871+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often the agent on each host will gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"The first TCP port the agent will try to bind to. Up to 1000 subsequent ports will be attempted if this port cannot be bound","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this option can facilitate debugging daemons that encounter runtime problems.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"Fraction of total system memory to divide among autotuned daemons. Converged systems hosting both Ceph and compute should set a lower value; systems with more than 10 GiB of memory per OSD may benefit from a slightly larger value so that the abundance of RAM can be fully utilized","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self-signed certificates generated and signed by the cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently Podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_alloy":{"name":"container_image_alloy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/alloy:latest","min":"","max":"","enum_allowed":[],"desc":"Alloy container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.6","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"Seconds for which to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"Location of dashboards to include in Grafana","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"How frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether the hardware monitoring daemon be deployed on every host?","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at the debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of daemons per service per host: guard against mistakes and runaways","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"Max number of OSDs that will be drained simultaneously when OSDs are removed","long_desc":"","tags":[],"see_also":[]},"max_parallel_osd_upgrades":{"name":"max_parallel_osd_upgrades","type":"int","level":"advanced","flags":0,"default_value":"16","min":"","max":"","enum_allowed":[],"desc":"Maximum number of OSD daemons upgraded in parallel.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"Mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default IP address for RedFish API (OOB management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"Location of alerts to include in Prometheus Alertmanager","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry URL for login purposes. This is not the default registry.","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"Cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Customized SSH config file to distribute to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many sequential SSH connection liveness check failures before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often SSH connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"How frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use the cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest to ensure that all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":true,"error_string":"","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"healthcheck_history_max_entries":{"name":"healthcheck_history_max_entries","type":"int","level":"advanced","flags":1,"default_value":"1000","min":"","max":"","enum_allowed":[],"desc":"Maximum number of health check history entries to keep","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"prometheus_tls_secret_name":{"name":"prometheus_tls_secret_name","type":"str","level":"advanced","flags":0,"default_value":"rook-ceph-prometheus-server-tls","min":"","max":"","enum_allowed":[],"desc":"name of tls secret in k8s for prometheus","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.183:0","nonce":1156717346}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.183:0","nonce":2983685632}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.183:0","nonce":1051967470}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.183:0","nonce":2597147796}]}]} 2026-02-25T08:41:26.831 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-25T08:41:26.831 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-25T08:41:26.831 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:26.907 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:26.903+0000 7f34056cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:26.910 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:26.908+0000 7f34056cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.025 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:27.025 INFO:teuthology.orchestra.run.trial169.stdout:{"epoch":15,"fsid":"a3da792f-4239-4b1a-ad7d-525e6c05ee6a","created":"2026-02-25T08:41:13.109418+0000","modified":"2026-02-25T08:41:26.359303+0000","last_up_change":"2026-02-25T08:41:22.339931+0000","last_in_change":"2026-02-25T08:41:14.983015+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-25T08:41:22.703588+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2026-02-25T08:41:23.341634+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c2af3fce-0bbe-497c-8eb5-86ae390e5605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6806","nonce":548378417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6805","nonce":548378417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6808","nonce":548378417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6807","nonce":548378417}]},"public_addr":"10.20.193.169:6806/548378417","cluster_addr":"10.20.193.169:6805/548378417","heartbeat_back_addr":"10.20.193.169:6808/548378417","heartbeat_front_addr":"10.20.193.169:6807/548378417","state":["exists","up"]},{"osd":1,"uuid":"1c19cfc2-9b80-4f78-88a9-7d7477f24d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6800","nonce":1807020326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6801","nonce":1807020326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6803","nonce":1807020326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6802","nonce":1807020326}]},"public_addr":"10.20.193.169:6800/1807020326","cluster_addr":"10.20.193.169:6801/1807020326","heartbeat_back_addr":"10.20.193.169:6803/1807020326","heartbeat_front_addr":"10.20.193.169:6802/1807020326","state":["exists","up"]},{"osd":2,"uuid":"61930931-bbe8-4bf2-9b0a-eebae4148e87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6803","nonce":406720561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6802","nonce":406720561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6807","nonce":406720561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6804","nonce":406720561}]},"public_addr":"10.20.193.183:6803/406720561","cluster_addr":"10.20.193.183:6802/406720561","heartbeat_back_addr":"10.20.193.183:6807/406720561","heartbeat_front_addr":"10.20.193.183:6804/406720561","state":["exists","up"]},{"osd":3,"uuid":"02f3ca2c-35c9-491e-9310-1a74c4aa8c04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6805","nonce":1517795374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6806","nonce":1517795374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6809","nonce":1517795374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6808","nonce":1517795374}]},"public_addr":"10.20.193.183:6805/1517795374","cluster_addr":"10.20.193.183:6806/1517795374","heartbeat_back_addr":"10.20.193.183:6809/1517795374","heartbeat_front_addr":"10.20.193.183:6808/1517795374","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:27.035 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-25T08:41:27.036 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:41:27.112 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.108+0000 7fc08645a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.114 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.112+0000 7fc08645a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.226 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:27.226 INFO:teuthology.orchestra.run.trial169.stdout:{"epoch":15,"fsid":"a3da792f-4239-4b1a-ad7d-525e6c05ee6a","created":"2026-02-25T08:41:13.109418+0000","modified":"2026-02-25T08:41:26.359303+0000","last_up_change":"2026-02-25T08:41:22.339931+0000","last_in_change":"2026-02-25T08:41:14.983015+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":true,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2026-02-25T08:41:22.703588+0000","flags":270337,"flags_names":"hashpspool,selfmanaged_snaps,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2026-02-25T08:41:23.341634+0000","flags":262145,"flags_names":"hashpspool,crimson","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.75,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c2af3fce-0bbe-497c-8eb5-86ae390e5605","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6806","nonce":548378417}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6805","nonce":548378417}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6808","nonce":548378417}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6807","nonce":548378417}]},"public_addr":"10.20.193.169:6806/548378417","cluster_addr":"10.20.193.169:6805/548378417","heartbeat_back_addr":"10.20.193.169:6808/548378417","heartbeat_front_addr":"10.20.193.169:6807/548378417","state":["exists","up"]},{"osd":1,"uuid":"1c19cfc2-9b80-4f78-88a9-7d7477f24d4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6800","nonce":1807020326}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6801","nonce":1807020326}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6803","nonce":1807020326}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.169:6802","nonce":1807020326}]},"public_addr":"10.20.193.169:6800/1807020326","cluster_addr":"10.20.193.169:6801/1807020326","heartbeat_back_addr":"10.20.193.169:6803/1807020326","heartbeat_front_addr":"10.20.193.169:6802/1807020326","state":["exists","up"]},{"osd":2,"uuid":"61930931-bbe8-4bf2-9b0a-eebae4148e87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6803","nonce":406720561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6802","nonce":406720561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6807","nonce":406720561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6804","nonce":406720561}]},"public_addr":"10.20.193.183:6803/406720561","cluster_addr":"10.20.193.183:6802/406720561","heartbeat_back_addr":"10.20.193.183:6807/406720561","heartbeat_front_addr":"10.20.193.183:6804/406720561","state":["exists","up"]},{"osd":3,"uuid":"02f3ca2c-35c9-491e-9310-1a74c4aa8c04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6805","nonce":1517795374}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6806","nonce":1517795374}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6809","nonce":1517795374}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.183:6808","nonce":1517795374}]},"public_addr":"10.20.193.183:6805/1517795374","cluster_addr":"10.20.193.183:6806/1517795374","heartbeat_back_addr":"10.20.193.183:6809/1517795374","heartbeat_front_addr":"10.20.193.183:6808/1517795374","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-25T08:41:27.236 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-25T08:41:27.236 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-25T08:41:27.236 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-25T08:41:27.236 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-25T08:41:27.319 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.315+0000 7fbfe0e3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.319 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.315+0000 7f21e421a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.319 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.315+0000 7f1d7585a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.319 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.315+0000 7fcb2a59c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.321 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.320+0000 7fbfe0e3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.322 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.320+0000 7f1d7585a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.322 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.320+0000 7f21e421a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.322 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.320+0000 7fcb2a59c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.339 INFO:teuthology.orchestra.run.trial169.stdout:47244640260 2026-02-25T08:41:27.339 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-25T08:41:27.340 INFO:teuthology.orchestra.run.trial169.stdout:42949672965 2026-02-25T08:41:27.340 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-25T08:41:27.341 INFO:teuthology.orchestra.run.trial169.stdout:47244640260 2026-02-25T08:41:27.341 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-25T08:41:27.343 INFO:teuthology.orchestra.run.trial169.stdout:47244640260 2026-02-25T08:41:27.343 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-25T08:41:27.421 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.417+0000 7fcb5821a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.421 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.417+0000 7f8351cc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.424 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.423+0000 7fcb5821a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.424 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.423+0000 7f8351cc2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.464 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.460+0000 7f8f73c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.465 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.461+0000 7fa514f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.467 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.465+0000 7f8f73c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.467 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.466+0000 7fa514f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.542 INFO:teuthology.orchestra.run.trial169.stdout:47244640260 2026-02-25T08:41:27.546 INFO:teuthology.orchestra.run.trial169.stdout:42949672965 2026-02-25T08:41:27.553 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.0 2026-02-25T08:41:27.553 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:27.557 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672965 got 42949672965 for osd.1 2026-02-25T08:41:27.558 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:27.582 INFO:teuthology.orchestra.run.trial169.stdout:47244640260 2026-02-25T08:41:27.588 INFO:teuthology.orchestra.run.trial169.stdout:47244640260 2026-02-25T08:41:27.593 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2026-02-25T08:41:27.593 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:27.598 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.3 2026-02-25T08:41:27.598 DEBUG:teuthology.parallel:result is None 2026-02-25T08:41:27.598 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T08:41:27.598 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:27.716 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.712+0000 7f987810b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.719 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.717+0000 7f987810b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.831 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:27.831 INFO:teuthology.orchestra.run.trial169.stderr:dumped all 2026-02-25T08:41:27.840 INFO:teuthology.orchestra.run.trial169.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-25T08:41:27.334929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890040,"kb_used_data":83890040,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825096,"statfs":{"total":214748299264,"available":128844898304,"internally_reserved":0,"allocated":85903400960,"data_stored":85903400960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.373011+0000","last_change":"2026-02-25T08:41:26.377529+0000","last_active":"2026-02-25T08:41:26.373011+0000","last_peered":"2026-02-25T08:41:26.373011+0000","last_clean":"2026-02-25T08:41:26.373011+0000","last_became_active":"2026-02-25T08:41:24.426559+0000","last_became_peered":"2026-02-25T08:41:24.426559+0000","last_unstale":"2026-02-25T08:41:26.373011+0000","last_undegraded":"2026-02-25T08:41:26.373011+0000","last_fullsized":"2026-02-25T08:41:26.373011+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.372633+0000","last_change":"2026-02-25T08:41:26.377253+0000","last_active":"2026-02-25T08:41:26.372633+0000","last_peered":"2026-02-25T08:41:26.372633+0000","last_clean":"2026-02-25T08:41:26.372633+0000","last_became_active":"2026-02-25T08:41:24.409565+0000","last_became_peered":"2026-02-25T08:41:24.409565+0000","last_unstale":"2026-02-25T08:41:26.372633+0000","last_undegraded":"2026-02-25T08:41:26.372633+0000","last_fullsized":"2026-02-25T08:41:26.372633+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.372788+0000","last_change":"2026-02-25T08:41:26.375854+0000","last_active":"2026-02-25T08:41:26.372788+0000","last_peered":"2026-02-25T08:41:26.372788+0000","last_clean":"2026-02-25T08:41:26.372788+0000","last_became_active":"2026-02-25T08:41:24.420122+0000","last_became_peered":"2026-02-25T08:41:24.420122+0000","last_unstale":"2026-02-25T08:41:26.372788+0000","last_undegraded":"2026-02-25T08:41:26.372788+0000","last_fullsized":"2026-02-25T08:41:26.372788+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.369685+0000","last_change":"2026-02-25T08:41:26.371743+0000","last_active":"2026-02-25T08:41:26.369685+0000","last_peered":"2026-02-25T08:41:26.369685+0000","last_clean":"2026-02-25T08:41:26.369685+0000","last_became_active":"2026-02-25T08:41:24.408398+0000","last_became_peered":"2026-02-25T08:41:24.408398+0000","last_unstale":"2026-02-25T08:41:26.369685+0000","last_undegraded":"2026-02-25T08:41:26.369685+0000","last_fullsized":"2026-02-25T08:41:26.369685+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.370335+0000","last_change":"2026-02-25T08:41:24.427616+0000","last_active":"2026-02-25T08:41:26.370335+0000","last_peered":"2026-02-25T08:41:26.370335+0000","last_clean":"2026-02-25T08:41:26.370335+0000","last_became_active":"2026-02-25T08:41:24.406166+0000","last_became_peered":"2026-02-25T08:41:24.406166+0000","last_unstale":"2026-02-25T08:41:26.370335+0000","last_undegraded":"2026-02-25T08:41:26.370335+0000","last_fullsized":"2026-02-25T08:41:26.370335+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.368219+0000","last_change":"2026-02-25T08:41:26.371067+0000","last_active":"2026-02-25T08:41:26.368219+0000","last_peered":"2026-02-25T08:41:26.368219+0000","last_clean":"2026-02-25T08:41:26.368219+0000","last_became_active":"2026-02-25T08:41:24.403313+0000","last_became_peered":"2026-02-25T08:41:24.403313+0000","last_unstale":"2026-02-25T08:41:26.368219+0000","last_undegraded":"2026-02-25T08:41:26.368219+0000","last_fullsized":"2026-02-25T08:41:26.368219+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.374022+0000","last_change":"2026-02-25T08:41:26.374022+0000","last_active":"2026-02-25T08:41:26.374022+0000","last_peered":"2026-02-25T08:41:26.374022+0000","last_clean":"2026-02-25T08:41:26.374022+0000","last_became_active":"2026-02-25T08:41:24.420798+0000","last_became_peered":"2026-02-25T08:41:24.420798+0000","last_unstale":"2026-02-25T08:41:26.374022+0000","last_undegraded":"2026-02-25T08:41:26.374022+0000","last_fullsized":"2026-02-25T08:41:26.374022+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.372524+0000","last_change":"2026-02-25T08:41:26.374903+0000","last_active":"2026-02-25T08:41:26.372524+0000","last_peered":"2026-02-25T08:41:26.372524+0000","last_clean":"2026-02-25T08:41:26.372524+0000","last_became_active":"2026-02-25T08:41:24.402449+0000","last_became_peered":"2026-02-25T08:41:24.402449+0000","last_unstale":"2026-02-25T08:41:26.372524+0000","last_undegraded":"2026-02-25T08:41:26.372524+0000","last_fullsized":"2026-02-25T08:41:26.372524+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.371094+0000","last_change":"2026-02-25T08:41:26.373872+0000","last_active":"2026-02-25T08:41:26.371094+0000","last_peered":"2026-02-25T08:41:26.371094+0000","last_clean":"2026-02-25T08:41:26.371094+0000","last_became_active":"2026-02-25T08:41:24.399215+0000","last_became_peered":"2026-02-25T08:41:24.399215+0000","last_unstale":"2026-02-25T08:41:26.371094+0000","last_undegraded":"2026-02-25T08:41:26.371094+0000","last_fullsized":"2026-02-25T08:41:26.371094+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210956288,"internally_reserved":0,"allocated":21476118528,"data_stored":21476118528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633810944,"internally_reserved":0,"allocated":64427413504,"data_stored":64427413504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972764,"kb_used_data":20972764,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456020,"statfs":{"total":53687074816,"available":32210964480,"internally_reserved":0,"allocated":21476110336,"data_stored":21476110336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971976,"kb_used_data":20971976,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456808,"statfs":{"total":53687074816,"available":32211771392,"internally_reserved":0,"allocated":21475303424,"data_stored":21475303424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972632,"kb_used_data":20972632,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456152,"statfs":{"total":53687074816,"available":32211099648,"internally_reserved":0,"allocated":21475975168,"data_stored":21475975168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972668,"kb_used_data":20972668,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456116,"statfs":{"total":53687074816,"available":32211062784,"internally_reserved":0,"allocated":21476012032,"data_stored":21476012032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211091456,"internally_reserved":0,"allocated":21475983360,"data_stored":21475983360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32210956288,"internally_reserved":0,"allocated":21476118528,"data_stored":21476118528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32210956288,"internally_reserved":0,"allocated":21476118528,"data_stored":21476118528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:27.841 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:27.918 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.914+0000 7fa0fefad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:27.920 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:27.919+0000 7fa0fefad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:28.029 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:28.029 INFO:teuthology.orchestra.run.trial169.stderr:dumped all 2026-02-25T08:41:28.038 INFO:teuthology.orchestra.run.trial169.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-25T08:41:27.334929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890040,"kb_used_data":83890040,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825096,"statfs":{"total":214748299264,"available":128844898304,"internally_reserved":0,"allocated":85903400960,"data_stored":85903400960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.373011+0000","last_change":"2026-02-25T08:41:26.377529+0000","last_active":"2026-02-25T08:41:26.373011+0000","last_peered":"2026-02-25T08:41:26.373011+0000","last_clean":"2026-02-25T08:41:26.373011+0000","last_became_active":"2026-02-25T08:41:24.426559+0000","last_became_peered":"2026-02-25T08:41:24.426559+0000","last_unstale":"2026-02-25T08:41:26.373011+0000","last_undegraded":"2026-02-25T08:41:26.373011+0000","last_fullsized":"2026-02-25T08:41:26.373011+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.372633+0000","last_change":"2026-02-25T08:41:26.377253+0000","last_active":"2026-02-25T08:41:26.372633+0000","last_peered":"2026-02-25T08:41:26.372633+0000","last_clean":"2026-02-25T08:41:26.372633+0000","last_became_active":"2026-02-25T08:41:24.409565+0000","last_became_peered":"2026-02-25T08:41:24.409565+0000","last_unstale":"2026-02-25T08:41:26.372633+0000","last_undegraded":"2026-02-25T08:41:26.372633+0000","last_fullsized":"2026-02-25T08:41:26.372633+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.372788+0000","last_change":"2026-02-25T08:41:26.375854+0000","last_active":"2026-02-25T08:41:26.372788+0000","last_peered":"2026-02-25T08:41:26.372788+0000","last_clean":"2026-02-25T08:41:26.372788+0000","last_became_active":"2026-02-25T08:41:24.420122+0000","last_became_peered":"2026-02-25T08:41:24.420122+0000","last_unstale":"2026-02-25T08:41:26.372788+0000","last_undegraded":"2026-02-25T08:41:26.372788+0000","last_fullsized":"2026-02-25T08:41:26.372788+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.369685+0000","last_change":"2026-02-25T08:41:26.371743+0000","last_active":"2026-02-25T08:41:26.369685+0000","last_peered":"2026-02-25T08:41:26.369685+0000","last_clean":"2026-02-25T08:41:26.369685+0000","last_became_active":"2026-02-25T08:41:24.408398+0000","last_became_peered":"2026-02-25T08:41:24.408398+0000","last_unstale":"2026-02-25T08:41:26.369685+0000","last_undegraded":"2026-02-25T08:41:26.369685+0000","last_fullsized":"2026-02-25T08:41:26.369685+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.370335+0000","last_change":"2026-02-25T08:41:24.427616+0000","last_active":"2026-02-25T08:41:26.370335+0000","last_peered":"2026-02-25T08:41:26.370335+0000","last_clean":"2026-02-25T08:41:26.370335+0000","last_became_active":"2026-02-25T08:41:24.406166+0000","last_became_peered":"2026-02-25T08:41:24.406166+0000","last_unstale":"2026-02-25T08:41:26.370335+0000","last_undegraded":"2026-02-25T08:41:26.370335+0000","last_fullsized":"2026-02-25T08:41:26.370335+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.368219+0000","last_change":"2026-02-25T08:41:26.371067+0000","last_active":"2026-02-25T08:41:26.368219+0000","last_peered":"2026-02-25T08:41:26.368219+0000","last_clean":"2026-02-25T08:41:26.368219+0000","last_became_active":"2026-02-25T08:41:24.403313+0000","last_became_peered":"2026-02-25T08:41:24.403313+0000","last_unstale":"2026-02-25T08:41:26.368219+0000","last_undegraded":"2026-02-25T08:41:26.368219+0000","last_fullsized":"2026-02-25T08:41:26.368219+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.374022+0000","last_change":"2026-02-25T08:41:26.374022+0000","last_active":"2026-02-25T08:41:26.374022+0000","last_peered":"2026-02-25T08:41:26.374022+0000","last_clean":"2026-02-25T08:41:26.374022+0000","last_became_active":"2026-02-25T08:41:24.420798+0000","last_became_peered":"2026-02-25T08:41:24.420798+0000","last_unstale":"2026-02-25T08:41:26.374022+0000","last_undegraded":"2026-02-25T08:41:26.374022+0000","last_fullsized":"2026-02-25T08:41:26.374022+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.372524+0000","last_change":"2026-02-25T08:41:26.374903+0000","last_active":"2026-02-25T08:41:26.372524+0000","last_peered":"2026-02-25T08:41:26.372524+0000","last_clean":"2026-02-25T08:41:26.372524+0000","last_became_active":"2026-02-25T08:41:24.402449+0000","last_became_peered":"2026-02-25T08:41:24.402449+0000","last_unstale":"2026-02-25T08:41:26.372524+0000","last_undegraded":"2026-02-25T08:41:26.372524+0000","last_fullsized":"2026-02-25T08:41:26.372524+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.371094+0000","last_change":"2026-02-25T08:41:26.373872+0000","last_active":"2026-02-25T08:41:26.371094+0000","last_peered":"2026-02-25T08:41:26.371094+0000","last_clean":"2026-02-25T08:41:26.371094+0000","last_became_active":"2026-02-25T08:41:24.399215+0000","last_became_peered":"2026-02-25T08:41:24.399215+0000","last_unstale":"2026-02-25T08:41:26.371094+0000","last_undegraded":"2026-02-25T08:41:26.371094+0000","last_fullsized":"2026-02-25T08:41:26.371094+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210956288,"internally_reserved":0,"allocated":21476118528,"data_stored":21476118528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633810944,"internally_reserved":0,"allocated":64427413504,"data_stored":64427413504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972764,"kb_used_data":20972764,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456020,"statfs":{"total":53687074816,"available":32210964480,"internally_reserved":0,"allocated":21476110336,"data_stored":21476110336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971976,"kb_used_data":20971976,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456808,"statfs":{"total":53687074816,"available":32211771392,"internally_reserved":0,"allocated":21475303424,"data_stored":21475303424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972632,"kb_used_data":20972632,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456152,"statfs":{"total":53687074816,"available":32211099648,"internally_reserved":0,"allocated":21475975168,"data_stored":21475975168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972668,"kb_used_data":20972668,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456116,"statfs":{"total":53687074816,"available":32211062784,"internally_reserved":0,"allocated":21476012032,"data_stored":21476012032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211091456,"internally_reserved":0,"allocated":21475983360,"data_stored":21475983360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32210956288,"internally_reserved":0,"allocated":21476118528,"data_stored":21476118528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32210956288,"internally_reserved":0,"allocated":21476118528,"data_stored":21476118528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:28.039 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T08:41:28.040 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-25T08:41:28.040 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-25T08:41:28.040 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-25T08:41:28.117 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:28.114+0000 7f045e05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:28.119 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:28.118+0000 7f045e05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:28.260 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:28.261 INFO:teuthology.orchestra.run.trial169.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-25T08:41:28.270 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-25T08:41:28.271 INFO:teuthology.run_tasks:Running task thrashosds... 2026-02-25T08:41:28.288 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 3, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2026-02-25T08:41:28.289 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}} 2026-02-25T08:41:28.289 INFO:tasks.thrashosds:config is {'ceph_objectstore_tool': False, 'chance_inject_pause_short': 0, 'chance_reset_purged_snaps_last': 0, 'chance_thrash_cluster_full': 0, 'chance_thrash_pg_upmap': 0, 'chance_trim_stale_osdmaps': 0, 'dump_ops_enable': False, 'min_in': 4, 'noscrub_toggle_delay': 0, 'reweight_osd': 0, 'sighup_delay': 0, 'thrash_primary_affinity': False, 'timeout': 2400, 'optrack_toggle_delay': 2.0, 'random_eio': 0.0} 2026-02-25T08:41:28.289 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2026-02-25T08:41:28.368 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:28.362+0000 7f44766cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:28.370 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:28.369+0000 7f44766cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:28.494 INFO:tasks.thrashosds:Beginning thrashosds... 2026-02-25T08:41:28.495 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T08:41:28.495 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:28.570 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:28.566+0000 7f82fc7b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:28.572 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:28.571+0000 7f82fc7b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:28.686 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:28.687 INFO:teuthology.orchestra.run.trial169.stderr:dumped all 2026-02-25T08:41:28.697 INFO:teuthology.orchestra.run.trial169.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-25T08:41:27.334929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890040,"kb_used_data":83890040,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825096,"statfs":{"total":214748299264,"available":128844898304,"internally_reserved":0,"allocated":85903400960,"data_stored":85903400960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.373011+0000","last_change":"2026-02-25T08:41:26.377529+0000","last_active":"2026-02-25T08:41:26.373011+0000","last_peered":"2026-02-25T08:41:26.373011+0000","last_clean":"2026-02-25T08:41:26.373011+0000","last_became_active":"2026-02-25T08:41:24.426559+0000","last_became_peered":"2026-02-25T08:41:24.426559+0000","last_unstale":"2026-02-25T08:41:26.373011+0000","last_undegraded":"2026-02-25T08:41:26.373011+0000","last_fullsized":"2026-02-25T08:41:26.373011+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.372633+0000","last_change":"2026-02-25T08:41:26.377253+0000","last_active":"2026-02-25T08:41:26.372633+0000","last_peered":"2026-02-25T08:41:26.372633+0000","last_clean":"2026-02-25T08:41:26.372633+0000","last_became_active":"2026-02-25T08:41:24.409565+0000","last_became_peered":"2026-02-25T08:41:24.409565+0000","last_unstale":"2026-02-25T08:41:26.372633+0000","last_undegraded":"2026-02-25T08:41:26.372633+0000","last_fullsized":"2026-02-25T08:41:26.372633+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.372788+0000","last_change":"2026-02-25T08:41:26.375854+0000","last_active":"2026-02-25T08:41:26.372788+0000","last_peered":"2026-02-25T08:41:26.372788+0000","last_clean":"2026-02-25T08:41:26.372788+0000","last_became_active":"2026-02-25T08:41:24.420122+0000","last_became_peered":"2026-02-25T08:41:24.420122+0000","last_unstale":"2026-02-25T08:41:26.372788+0000","last_undegraded":"2026-02-25T08:41:26.372788+0000","last_fullsized":"2026-02-25T08:41:26.372788+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.369685+0000","last_change":"2026-02-25T08:41:26.371743+0000","last_active":"2026-02-25T08:41:26.369685+0000","last_peered":"2026-02-25T08:41:26.369685+0000","last_clean":"2026-02-25T08:41:26.369685+0000","last_became_active":"2026-02-25T08:41:24.408398+0000","last_became_peered":"2026-02-25T08:41:24.408398+0000","last_unstale":"2026-02-25T08:41:26.369685+0000","last_undegraded":"2026-02-25T08:41:26.369685+0000","last_fullsized":"2026-02-25T08:41:26.369685+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.370335+0000","last_change":"2026-02-25T08:41:24.427616+0000","last_active":"2026-02-25T08:41:26.370335+0000","last_peered":"2026-02-25T08:41:26.370335+0000","last_clean":"2026-02-25T08:41:26.370335+0000","last_became_active":"2026-02-25T08:41:24.406166+0000","last_became_peered":"2026-02-25T08:41:24.406166+0000","last_unstale":"2026-02-25T08:41:26.370335+0000","last_undegraded":"2026-02-25T08:41:26.370335+0000","last_fullsized":"2026-02-25T08:41:26.370335+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.368219+0000","last_change":"2026-02-25T08:41:26.371067+0000","last_active":"2026-02-25T08:41:26.368219+0000","last_peered":"2026-02-25T08:41:26.368219+0000","last_clean":"2026-02-25T08:41:26.368219+0000","last_became_active":"2026-02-25T08:41:24.403313+0000","last_became_peered":"2026-02-25T08:41:24.403313+0000","last_unstale":"2026-02-25T08:41:26.368219+0000","last_undegraded":"2026-02-25T08:41:26.368219+0000","last_fullsized":"2026-02-25T08:41:26.368219+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.374022+0000","last_change":"2026-02-25T08:41:26.374022+0000","last_active":"2026-02-25T08:41:26.374022+0000","last_peered":"2026-02-25T08:41:26.374022+0000","last_clean":"2026-02-25T08:41:26.374022+0000","last_became_active":"2026-02-25T08:41:24.420798+0000","last_became_peered":"2026-02-25T08:41:24.420798+0000","last_unstale":"2026-02-25T08:41:26.374022+0000","last_undegraded":"2026-02-25T08:41:26.374022+0000","last_fullsized":"2026-02-25T08:41:26.374022+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.372524+0000","last_change":"2026-02-25T08:41:26.374903+0000","last_active":"2026-02-25T08:41:26.372524+0000","last_peered":"2026-02-25T08:41:26.372524+0000","last_clean":"2026-02-25T08:41:26.372524+0000","last_became_active":"2026-02-25T08:41:24.402449+0000","last_became_peered":"2026-02-25T08:41:24.402449+0000","last_unstale":"2026-02-25T08:41:26.372524+0000","last_undegraded":"2026-02-25T08:41:26.372524+0000","last_fullsized":"2026-02-25T08:41:26.372524+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.371094+0000","last_change":"2026-02-25T08:41:26.373872+0000","last_active":"2026-02-25T08:41:26.371094+0000","last_peered":"2026-02-25T08:41:26.371094+0000","last_clean":"2026-02-25T08:41:26.371094+0000","last_became_active":"2026-02-25T08:41:24.399215+0000","last_became_peered":"2026-02-25T08:41:24.399215+0000","last_unstale":"2026-02-25T08:41:26.371094+0000","last_undegraded":"2026-02-25T08:41:26.371094+0000","last_fullsized":"2026-02-25T08:41:26.371094+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210956288,"internally_reserved":0,"allocated":21476118528,"data_stored":21476118528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633810944,"internally_reserved":0,"allocated":64427413504,"data_stored":64427413504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972764,"kb_used_data":20972764,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456020,"statfs":{"total":53687074816,"available":32210964480,"internally_reserved":0,"allocated":21476110336,"data_stored":21476110336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971976,"kb_used_data":20971976,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456808,"statfs":{"total":53687074816,"available":32211771392,"internally_reserved":0,"allocated":21475303424,"data_stored":21475303424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972632,"kb_used_data":20972632,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456152,"statfs":{"total":53687074816,"available":32211099648,"internally_reserved":0,"allocated":21475975168,"data_stored":21475975168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972668,"kb_used_data":20972668,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456116,"statfs":{"total":53687074816,"available":32211062784,"internally_reserved":0,"allocated":21476012032,"data_stored":21476012032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211091456,"internally_reserved":0,"allocated":21475983360,"data_stored":21475983360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32210956288,"internally_reserved":0,"allocated":21476118528,"data_stored":21476118528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32210956288,"internally_reserved":0,"allocated":21476118528,"data_stored":21476118528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:28.699 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:28.776 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:28.773+0000 7fa5ba5dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:28.779 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:28.777+0000 7fa5ba5dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:28.890 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:28.890 INFO:teuthology.orchestra.run.trial169.stderr:dumped all 2026-02-25T08:41:28.900 INFO:teuthology.orchestra.run.trial169.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2026-02-25T08:41:27.334929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54,"ondisk_log_size":54,"up":27,"acting":27,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83890040,"kb_used_data":83890040,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125825096,"statfs":{"total":214748299264,"available":128844898304,"internally_reserved":0,"allocated":85903400960,"data_stored":85903400960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.373011+0000","last_change":"2026-02-25T08:41:26.377529+0000","last_active":"2026-02-25T08:41:26.373011+0000","last_peered":"2026-02-25T08:41:26.373011+0000","last_clean":"2026-02-25T08:41:26.373011+0000","last_became_active":"2026-02-25T08:41:24.426559+0000","last_became_peered":"2026-02-25T08:41:24.426559+0000","last_unstale":"2026-02-25T08:41:26.373011+0000","last_undegraded":"2026-02-25T08:41:26.373011+0000","last_fullsized":"2026-02-25T08:41:26.373011+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.372633+0000","last_change":"2026-02-25T08:41:26.377253+0000","last_active":"2026-02-25T08:41:26.372633+0000","last_peered":"2026-02-25T08:41:26.372633+0000","last_clean":"2026-02-25T08:41:26.372633+0000","last_became_active":"2026-02-25T08:41:24.409565+0000","last_became_peered":"2026-02-25T08:41:24.409565+0000","last_unstale":"2026-02-25T08:41:26.372633+0000","last_undegraded":"2026-02-25T08:41:26.372633+0000","last_fullsized":"2026-02-25T08:41:26.372633+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.372788+0000","last_change":"2026-02-25T08:41:26.375854+0000","last_active":"2026-02-25T08:41:26.372788+0000","last_peered":"2026-02-25T08:41:26.372788+0000","last_clean":"2026-02-25T08:41:26.372788+0000","last_became_active":"2026-02-25T08:41:24.420122+0000","last_became_peered":"2026-02-25T08:41:24.420122+0000","last_unstale":"2026-02-25T08:41:26.372788+0000","last_undegraded":"2026-02-25T08:41:26.372788+0000","last_fullsized":"2026-02-25T08:41:26.372788+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.369685+0000","last_change":"2026-02-25T08:41:26.371743+0000","last_active":"2026-02-25T08:41:26.369685+0000","last_peered":"2026-02-25T08:41:26.369685+0000","last_clean":"2026-02-25T08:41:26.369685+0000","last_became_active":"2026-02-25T08:41:24.408398+0000","last_became_peered":"2026-02-25T08:41:24.408398+0000","last_unstale":"2026-02-25T08:41:26.369685+0000","last_undegraded":"2026-02-25T08:41:26.369685+0000","last_fullsized":"2026-02-25T08:41:26.369685+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":82,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.370335+0000","last_change":"2026-02-25T08:41:24.427616+0000","last_active":"2026-02-25T08:41:26.370335+0000","last_peered":"2026-02-25T08:41:26.370335+0000","last_clean":"2026-02-25T08:41:26.370335+0000","last_became_active":"2026-02-25T08:41:24.406166+0000","last_became_peered":"2026-02-25T08:41:24.406166+0000","last_unstale":"2026-02-25T08:41:26.370335+0000","last_undegraded":"2026-02-25T08:41:26.370335+0000","last_fullsized":"2026-02-25T08:41:26.370335+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.368219+0000","last_change":"2026-02-25T08:41:26.371067+0000","last_active":"2026-02-25T08:41:26.368219+0000","last_peered":"2026-02-25T08:41:26.368219+0000","last_clean":"2026-02-25T08:41:26.368219+0000","last_became_active":"2026-02-25T08:41:24.403313+0000","last_became_peered":"2026-02-25T08:41:24.403313+0000","last_unstale":"2026-02-25T08:41:26.368219+0000","last_undegraded":"2026-02-25T08:41:26.368219+0000","last_fullsized":"2026-02-25T08:41:26.368219+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"14'1","reported_seq":18,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.374022+0000","last_change":"2026-02-25T08:41:26.374022+0000","last_active":"2026-02-25T08:41:26.374022+0000","last_peered":"2026-02-25T08:41:26.374022+0000","last_clean":"2026-02-25T08:41:26.374022+0000","last_became_active":"2026-02-25T08:41:24.420798+0000","last_became_peered":"2026-02-25T08:41:24.420798+0000","last_unstale":"2026-02-25T08:41:26.374022+0000","last_undegraded":"2026-02-25T08:41:26.374022+0000","last_fullsized":"2026-02-25T08:41:26.374022+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.372524+0000","last_change":"2026-02-25T08:41:26.374903+0000","last_active":"2026-02-25T08:41:26.372524+0000","last_peered":"2026-02-25T08:41:26.372524+0000","last_clean":"2026-02-25T08:41:26.372524+0000","last_became_active":"2026-02-25T08:41:24.402449+0000","last_became_peered":"2026-02-25T08:41:24.402449+0000","last_unstale":"2026-02-25T08:41:26.372524+0000","last_undegraded":"2026-02-25T08:41:26.372524+0000","last_fullsized":"2026-02-25T08:41:26.372524+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":15,"state":"active+clean","last_fresh":"2026-02-25T08:41:26.371094+0000","last_change":"2026-02-25T08:41:26.373872+0000","last_active":"2026-02-25T08:41:26.371094+0000","last_peered":"2026-02-25T08:41:26.371094+0000","last_clean":"2026-02-25T08:41:26.371094+0000","last_became_active":"2026-02-25T08:41:24.399215+0000","last_became_peered":"2026-02-25T08:41:24.399215+0000","last_unstale":"2026-02-25T08:41:26.371094+0000","last_undegraded":"2026-02-25T08:41:26.371094+0000","last_fullsized":"2026-02-25T08:41:26.371094+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210956288,"internally_reserved":0,"allocated":21476118528,"data_stored":21476118528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96633810944,"internally_reserved":0,"allocated":64427413504,"data_stored":64427413504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972764,"kb_used_data":20972764,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456020,"statfs":{"total":53687074816,"available":32210964480,"internally_reserved":0,"allocated":21476110336,"data_stored":21476110336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20971976,"kb_used_data":20971976,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456808,"statfs":{"total":53687074816,"available":32211771392,"internally_reserved":0,"allocated":21475303424,"data_stored":21475303424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972632,"kb_used_data":20972632,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456152,"statfs":{"total":53687074816,"available":32211099648,"internally_reserved":0,"allocated":21475975168,"data_stored":21475975168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640260,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972668,"kb_used_data":20972668,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456116,"statfs":{"total":53687074816,"available":32211062784,"internally_reserved":0,"allocated":21476012032,"data_stored":21476012032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32211091456,"internally_reserved":0,"allocated":21475983360,"data_stored":21475983360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32211763200,"internally_reserved":0,"allocated":21475311616,"data_stored":21475311616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32210956288,"internally_reserved":0,"allocated":21476118528,"data_stored":21476118528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32210956288,"internally_reserved":0,"allocated":21476118528,"data_stored":21476118528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:28.900 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-25T08:41:28.900 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2026-02-25T08:41:28.978 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:28.974+0000 7f5046ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:28.980 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:28.979+0000 7f5046ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:29.093 INFO:teuthology.orchestra.run.trial169.stdout:epoch 15 2026-02-25T08:41:29.093 INFO:teuthology.orchestra.run.trial169.stdout:fsid a3da792f-4239-4b1a-ad7d-525e6c05ee6a 2026-02-25T08:41:29.094 INFO:teuthology.orchestra.run.trial169.stdout:created 2026-02-25T08:41:13.109418+0000 2026-02-25T08:41:29.094 INFO:teuthology.orchestra.run.trial169.stdout:modified 2026-02-25T08:41:26.359303+0000 2026-02-25T08:41:29.094 INFO:teuthology.orchestra.run.trial169.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2026-02-25T08:41:29.094 INFO:teuthology.orchestra.run.trial169.stdout:crush_version 5 2026-02-25T08:41:29.094 INFO:teuthology.orchestra.run.trial169.stdout:full_ratio 0.95 2026-02-25T08:41:29.094 INFO:teuthology.orchestra.run.trial169.stdout:backfillfull_ratio 0.9 2026-02-25T08:41:29.094 INFO:teuthology.orchestra.run.trial169.stdout:nearfull_ratio 0.85 2026-02-25T08:41:29.094 INFO:teuthology.orchestra.run.trial169.stdout:require_min_compat_client luminous 2026-02-25T08:41:29.094 INFO:teuthology.orchestra.run.trial169.stdout:min_compat_client jewel 2026-02-25T08:41:29.094 INFO:teuthology.orchestra.run.trial169.stdout:require_osd_release tentacle 2026-02-25T08:41:29.094 INFO:teuthology.orchestra.run.trial169.stdout:stretch_mode_enabled false 2026-02-25T08:41:29.094 INFO:teuthology.orchestra.run.trial169.stdout:allow_crimson=true 2026-02-25T08:41:29.095 INFO:teuthology.orchestra.run.trial169.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 15 flags hashpspool,selfmanaged_snaps,crimson stripe_width 0 application rbd read_balance_score 2.00 2026-02-25T08:41:29.095 INFO:teuthology.orchestra.run.trial169.stdout: removed_snaps_queue [2~1] 2026-02-25T08:41:29.095 INFO:teuthology.orchestra.run.trial169.stdout:pool 2 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 14 flags hashpspool,crimson stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 4.00 2026-02-25T08:41:29.095 INFO:teuthology.orchestra.run.trial169.stdout:max_osd 4 2026-02-25T08:41:29.095 INFO:teuthology.orchestra.run.trial169.stdout:osd.0 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) v2:10.20.193.169:6806/548378417 v2:10.20.193.169:6805/548378417 exists,up c2af3fce-0bbe-497c-8eb5-86ae390e5605 2026-02-25T08:41:29.095 INFO:teuthology.orchestra.run.trial169.stdout:osd.1 up in weight 1 up_from 10 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.169:6800/1807020326 v2:10.20.193.169:6801/1807020326 exists,up 1c19cfc2-9b80-4f78-88a9-7d7477f24d4f 2026-02-25T08:41:29.095 INFO:teuthology.orchestra.run.trial169.stdout:osd.2 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.183:6803/406720561 v2:10.20.193.183:6802/406720561 exists,up 61930931-bbe8-4bf2-9b0a-eebae4148e87 2026-02-25T08:41:29.095 INFO:teuthology.orchestra.run.trial169.stdout:osd.3 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.183:6805/1517795374 v2:10.20.193.183:6806/1517795374 exists,up 02f3ca2c-35c9-491e-9310-1a74c4aa8c04 2026-02-25T08:41:29.103 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) v2:10.20.193.169:6806/548378417 v2:10.20.193.169:6805/548378417 exists,up c2af3fce-0bbe-497c-8eb5-86ae390e5605', 'osd.1 up in weight 1 up_from 10 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.169:6800/1807020326 v2:10.20.193.169:6801/1807020326 exists,up 1c19cfc2-9b80-4f78-88a9-7d7477f24d4f', 'osd.2 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.183:6803/406720561 v2:10.20.193.183:6802/406720561 exists,up 61930931-bbe8-4bf2-9b0a-eebae4148e87', 'osd.3 up in weight 1 up_from 11 up_thru 12 down_at 0 last_clean_interval [0,0) v2:10.20.193.183:6805/1517795374 v2:10.20.193.183:6806/1517795374 exists,up 02f3ca2c-35c9-491e-9310-1a74c4aa8c04'] 2026-02-25T08:41:29.103 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2026-02-25T08:41:29.179 INFO:teuthology.orchestra.run.trial169.stdout:{ 2026-02-25T08:41:29.179 INFO:teuthology.orchestra.run.trial169.stdout: "name": "mon.a", 2026-02-25T08:41:29.179 INFO:teuthology.orchestra.run.trial169.stdout: "cluster": "ceph", 2026-02-25T08:41:29.179 INFO:teuthology.orchestra.run.trial169.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2026-02-25T08:41:29.179 INFO:teuthology.orchestra.run.trial169.stdout: "admin_socket_mode": "", 2026-02-25T08:41:29.179 INFO:teuthology.orchestra.run.trial169.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2026-02-25T08:41:29.179 INFO:teuthology.orchestra.run.trial169.stdout: "auth_client_required": "cephx, none", 2026-02-25T08:41:29.180 INFO:teuthology.orchestra.run.trial169.stdout: "auth_cluster_required": "cephx", 2026-02-25T08:41:29.180 INFO:teuthology.orchestra.run.trial169.stdout: "auth_debug": "true", 2026-02-25T08:41:29.180 INFO:teuthology.orchestra.run.trial169.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2026-02-25T08:41:29.180 INFO:teuthology.orchestra.run.trial169.stdout: "auth_mon_ticket_ttl": "660.000000", 2026-02-25T08:41:29.180 INFO:teuthology.orchestra.run.trial169.stdout: "auth_service_required": "cephx", 2026-02-25T08:41:29.180 INFO:teuthology.orchestra.run.trial169.stdout: "auth_service_ticket_ttl": "240.000000", 2026-02-25T08:41:29.180 INFO:teuthology.orchestra.run.trial169.stdout: "auth_supported": "cephx", 2026-02-25T08:41:29.180 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_aio": "true", 2026-02-25T08:41:29.180 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_aio_max_queue_depth": "1024", 2026-02-25T08:41:29.180 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_aio_poll_ms": "250", 2026-02-25T08:41:29.180 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_aio_reap_max": "16", 2026-02-25T08:41:29.180 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_aio_submit_retry_initial_delay_us": "125", 2026-02-25T08:41:29.181 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_aio_submit_retry_max": "16", 2026-02-25T08:41:29.181 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_async_discard": "false", 2026-02-25T08:41:29.181 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_async_discard_max_pending": "1000000", 2026-02-25T08:41:29.181 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_async_discard_threads": "0", 2026-02-25T08:41:29.181 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_block_size": "4096", 2026-02-25T08:41:29.181 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_debug_aio": "false", 2026-02-25T08:41:29.181 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_debug_aio_log_age": "5.000000", 2026-02-25T08:41:29.181 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2026-02-25T08:41:29.181 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_debug_discard_sleep": "0", 2026-02-25T08:41:29.181 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_debug_inflight_ios": "false", 2026-02-25T08:41:29.181 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_discard_max_bytes": "10737418240", 2026-02-25T08:41:29.181 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_enable_discard": "false", 2026-02-25T08:41:29.182 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_flock_retry": "3", 2026-02-25T08:41:29.182 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_flock_retry_interval": "0.100000", 2026-02-25T08:41:29.182 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_inject_crash": "0", 2026-02-25T08:41:29.182 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_inject_crash_flush_delay": "2", 2026-02-25T08:41:29.182 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_ioring": "false", 2026-02-25T08:41:29.182 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_ioring_hipri": "false", 2026-02-25T08:41:29.182 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_ioring_sqthread_poll": "false", 2026-02-25T08:41:29.182 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_max_discard_length": "2147483648", 2026-02-25T08:41:29.182 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_nvme_unbind_from_kernel": "false", 2026-02-25T08:41:29.182 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_read_buffer_alignment": "4096", 2026-02-25T08:41:29.182 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_read_preallocated_huge_buffers": "", 2026-02-25T08:41:29.182 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2026-02-25T08:41:29.183 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_stalled_read_warn_threshold": "1", 2026-02-25T08:41:29.183 INFO:teuthology.orchestra.run.trial169.stdout: "bdev_type": "", 2026-02-25T08:41:29.183 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_alloc_size": "1048576", 2026-02-25T08:41:29.183 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_allocator": "hybrid", 2026-02-25T08:41:29.183 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_buffered_io": "true", 2026-02-25T08:41:29.183 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_check_for_zeros": "false", 2026-02-25T08:41:29.183 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_check_volume_selector_often": "false", 2026-02-25T08:41:29.183 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_check_volume_selector_on_umount": "false", 2026-02-25T08:41:29.183 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_compact_log_sync": "false", 2026-02-25T08:41:29.183 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2026-02-25T08:41:29.183 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2026-02-25T08:41:29.183 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_log_compact_min_size": "16777216", 2026-02-25T08:41:29.183 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_log_replay_check_allocations": "true", 2026-02-25T08:41:29.184 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_max_log_runway": "4194304", 2026-02-25T08:41:29.184 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_max_prefetch": "1048576", 2026-02-25T08:41:29.184 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_min_flush_size": "524288", 2026-02-25T08:41:29.184 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_min_log_runway": "1048576", 2026-02-25T08:41:29.184 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_replay_recovery": "false", 2026-02-25T08:41:29.184 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_replay_recovery_disable_compact": "false", 2026-02-25T08:41:29.184 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_shared_alloc_size": "65536", 2026-02-25T08:41:29.184 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_sync_write": "false", 2026-02-25T08:41:29.184 INFO:teuthology.orchestra.run.trial169.stdout: "bluefs_wal_envelope_mode": "true", 2026-02-25T08:41:29.184 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2026-02-25T08:41:29.184 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2026-02-25T08:41:29.184 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2026-02-25T08:41:29.184 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_allocation_from_file": "true", 2026-02-25T08:41:29.185 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_allocator": "hybrid", 2026-02-25T08:41:29.185 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_allocator_lookup_policy": "auto", 2026-02-25T08:41:29.185 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_async_db_compaction": "true", 2026-02-25T08:41:29.185 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2026-02-25T08:41:29.185 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2026-02-25T08:41:29.185 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2026-02-25T08:41:29.185 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2026-02-25T08:41:29.185 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_bdev_label_multi": "true", 2026-02-25T08:41:29.185 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2026-02-25T08:41:29.185 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_bdev_label_require_all": "true", 2026-02-25T08:41:29.185 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2026-02-25T08:41:29.185 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_bitmapallocator_span_size": "1024", 2026-02-25T08:41:29.185 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_blobid_prealloc": "10240", 2026-02-25T08:41:29.185 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_block_create": "true", 2026-02-25T08:41:29.186 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_block_db_create": "false", 2026-02-25T08:41:29.186 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_block_db_path": "", 2026-02-25T08:41:29.186 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_block_db_size": "0", 2026-02-25T08:41:29.186 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_block_path": "", 2026-02-25T08:41:29.186 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_block_preallocate_file": "false", 2026-02-25T08:41:29.186 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_block_size": "107374182400", 2026-02-25T08:41:29.186 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_block_wal_create": "false", 2026-02-25T08:41:29.186 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_block_wal_path": "", 2026-02-25T08:41:29.186 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_block_wal_size": "100663296", 2026-02-25T08:41:29.186 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_bluefs": "true", 2026-02-25T08:41:29.186 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2026-02-25T08:41:29.186 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_bluefs_env_mirror": "false", 2026-02-25T08:41:29.187 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_bluefs_max_free": "10737418240", 2026-02-25T08:41:29.187 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_btree2_alloc_weight_factor": "2.000000", 2026-02-25T08:41:29.187 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2026-02-25T08:41:29.187 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2026-02-25T08:41:29.187 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2026-02-25T08:41:29.187 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2026-02-25T08:41:29.187 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2026-02-25T08:41:29.187 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_autotune": "true", 2026-02-25T08:41:29.187 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_autotune_interval": "5.000000", 2026-02-25T08:41:29.187 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2026-02-25T08:41:29.187 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_kv_ratio": "0.450000", 2026-02-25T08:41:29.187 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_meta_ratio": "0.450000", 2026-02-25T08:41:29.187 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_size": "0", 2026-02-25T08:41:29.188 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_size_hdd": "1073741824", 2026-02-25T08:41:29.188 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_size_ssd": "3221225472", 2026-02-25T08:41:29.188 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_trim_interval": "0.050000", 2026-02-25T08:41:29.188 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2026-02-25T08:41:29.188 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cache_type": "2q", 2026-02-25T08:41:29.188 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2026-02-25T08:41:29.188 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_clone_cow": "true", 2026-02-25T08:41:29.188 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_compression_algorithm": "snappy", 2026-02-25T08:41:29.188 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_compression_max_blob_size": "0", 2026-02-25T08:41:29.188 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2026-02-25T08:41:29.188 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2026-02-25T08:41:29.188 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_compression_min_blob_size": "0", 2026-02-25T08:41:29.188 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2026-02-25T08:41:29.188 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2026-02-25T08:41:29.189 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_compression_mode": "none", 2026-02-25T08:41:29.189 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_compression_required_ratio": "0.875000", 2026-02-25T08:41:29.189 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_csum_type": "crc32c", 2026-02-25T08:41:29.189 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2026-02-25T08:41:29.189 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_enforce_settings": "default", 2026-02-25T08:41:29.189 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_extent_map_encode_check": "false", 2026-02-25T08:41:29.189 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_freelist": "false", 2026-02-25T08:41:29.189 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_fsck_abort": "false", 2026-02-25T08:41:29.189 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2026-02-25T08:41:29.189 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2026-02-25T08:41:29.189 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_inject_read_err": "false", 2026-02-25T08:41:29.189 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_legacy_omap": "false", 2026-02-25T08:41:29.189 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_no_reuse_blocks": "false", 2026-02-25T08:41:29.190 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_omit_block_device_write": "false", 2026-02-25T08:41:29.190 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_omit_kv_commit": "false", 2026-02-25T08:41:29.190 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_onode_segmentation_random": "false", 2026-02-25T08:41:29.190 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2026-02-25T08:41:29.190 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_prefragment_max": "1048576", 2026-02-25T08:41:29.190 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_random_read_err": "0.000000", 2026-02-25T08:41:29.190 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2026-02-25T08:41:29.190 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_small_allocations": "0", 2026-02-25T08:41:29.190 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2026-02-25T08:41:29.190 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_default_buffered_read": "true", 2026-02-25T08:41:29.190 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_default_buffered_write": "false", 2026-02-25T08:41:29.190 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_deferred_batch_ops": "0", 2026-02-25T08:41:29.190 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2026-02-25T08:41:29.191 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2026-02-25T08:41:29.191 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_discard_on_mkfs": "true", 2026-02-25T08:41:29.191 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_elastic_shared_blobs": "true", 2026-02-25T08:41:29.191 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2026-02-25T08:41:29.191 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_extent_map_shard_max_size": "1200", 2026-02-25T08:41:29.191 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_extent_map_shard_min_size": "150", 2026-02-25T08:41:29.191 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_extent_map_shard_target_size": "500", 2026-02-25T08:41:29.191 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2026-02-25T08:41:29.191 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fail_eio": "false", 2026-02-25T08:41:29.191 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fragmentation_check_period": "3600", 2026-02-25T08:41:29.191 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_freelist_blocks_per_key": "128", 2026-02-25T08:41:29.191 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2026-02-25T08:41:29.191 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2026-02-25T08:41:29.191 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2026-02-25T08:41:29.192 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fsck_on_mkfs": "true", 2026-02-25T08:41:29.192 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2026-02-25T08:41:29.192 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fsck_on_mount": "false", 2026-02-25T08:41:29.192 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fsck_on_mount_deep": "false", 2026-02-25T08:41:29.192 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fsck_on_umount": "false", 2026-02-25T08:41:29.192 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fsck_on_umount_deep": "false", 2026-02-25T08:41:29.192 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2026-02-25T08:41:29.192 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fsck_quick_fix_threads": "2", 2026-02-25T08:41:29.192 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2026-02-25T08:41:29.192 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2026-02-25T08:41:29.192 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_gc_enable_blob_threshold": "0", 2026-02-25T08:41:29.192 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_gc_enable_total_threshold": "0", 2026-02-25T08:41:29.192 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2026-02-25T08:41:29.193 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_ignore_data_csum": "false", 2026-02-25T08:41:29.193 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2026-02-25T08:41:29.193 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_kvbackend": "rocksdb", 2026-02-25T08:41:29.193 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_log_collection_list_age": "60.000000", 2026-02-25T08:41:29.193 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2026-02-25T08:41:29.193 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_log_op_age": "5.000000", 2026-02-25T08:41:29.193 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_max_alloc_size": "0", 2026-02-25T08:41:29.193 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_max_blob_size": "0", 2026-02-25T08:41:29.193 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_max_blob_size_hdd": "65536", 2026-02-25T08:41:29.193 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_max_blob_size_ssd": "65536", 2026-02-25T08:41:29.193 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_max_defer_interval": "3.000000", 2026-02-25T08:41:29.193 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_max_deferred_txc": "32", 2026-02-25T08:41:29.194 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_min_alloc_size": "0", 2026-02-25T08:41:29.194 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_min_alloc_size_hdd": "4096", 2026-02-25T08:41:29.194 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_min_alloc_size_ssd": "4096", 2026-02-25T08:41:29.194 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_nid_prealloc": "1024", 2026-02-25T08:41:29.194 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_onode_segment_size": "0", 2026-02-25T08:41:29.194 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_prefer_deferred_size": "0", 2026-02-25T08:41:29.194 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2026-02-25T08:41:29.194 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2026-02-25T08:41:29.194 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_qfsck_on_mount": "true", 2026-02-25T08:41:29.194 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_recompression_min_gain": "1.200000", 2026-02-25T08:41:29.194 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_retry_disk_reads": "3", 2026-02-25T08:41:29.194 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_rocksdb_cf": "true", 2026-02-25T08:41:29.194 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2026-02-25T08:41:29.195 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2026-02-25T08:41:29.195 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_rocksdb_options_annex": "", 2026-02-25T08:41:29.195 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2026-02-25T08:41:29.195 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_slow_ops_warn_threshold": "1", 2026-02-25T08:41:29.195 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_spdk_coremask": "0x1", 2026-02-25T08:41:29.195 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_spdk_io_sleep": "5", 2026-02-25T08:41:29.195 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_spdk_max_io_completion": "0", 2026-02-25T08:41:29.195 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_spdk_mem": "512", 2026-02-25T08:41:29.195 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_sync_submit_transaction": "false", 2026-02-25T08:41:29.195 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_throttle_bytes": "67108864", 2026-02-25T08:41:29.195 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_throttle_cost_per_io": "0", 2026-02-25T08:41:29.195 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2026-02-25T08:41:29.195 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2026-02-25T08:41:29.196 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2026-02-25T08:41:29.196 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_throttle_trace_rate": "0.000000", 2026-02-25T08:41:29.196 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_tracing": "false", 2026-02-25T08:41:29.196 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2026-02-25T08:41:29.196 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2026-02-25T08:41:29.196 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_volume_selection_reserved": "0", 2026-02-25T08:41:29.196 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2026-02-25T08:41:29.196 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2026-02-25T08:41:29.196 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_warn_on_free_fragmentation": "0.800000", 2026-02-25T08:41:29.196 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_warn_on_legacy_statfs": "true", 2026-02-25T08:41:29.196 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2026-02-25T08:41:29.196 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2026-02-25T08:41:29.196 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2026-02-25T08:41:29.197 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_write_v2": "false", 2026-02-25T08:41:29.197 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_write_v2_random": "false", 2026-02-25T08:41:29.197 INFO:teuthology.orchestra.run.trial169.stdout: "bluestore_zero_block_detection": "false", 2026-02-25T08:41:29.197 INFO:teuthology.orchestra.run.trial169.stdout: "breakpad": "false", 2026-02-25T08:41:29.197 INFO:teuthology.orchestra.run.trial169.stdout: "ceph_assert_supresssions": "", 2026-02-25T08:41:29.197 INFO:teuthology.orchestra.run.trial169.stdout: "cephadm_path": "/usr/sbin/cephadm", 2026-02-25T08:41:29.197 INFO:teuthology.orchestra.run.trial169.stdout: "cephfs_mirror_action_update_interval": "2", 2026-02-25T08:41:29.197 INFO:teuthology.orchestra.run.trial169.stdout: "cephfs_mirror_directory_scan_interval": "10", 2026-02-25T08:41:29.197 INFO:teuthology.orchestra.run.trial169.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2026-02-25T08:41:29.197 INFO:teuthology.orchestra.run.trial169.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2026-02-25T08:41:29.197 INFO:teuthology.orchestra.run.trial169.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2026-02-25T08:41:29.197 INFO:teuthology.orchestra.run.trial169.stdout: "cephfs_mirror_mount_timeout": "10", 2026-02-25T08:41:29.197 INFO:teuthology.orchestra.run.trial169.stdout: "cephfs_mirror_perf_stats_prio": "5", 2026-02-25T08:41:29.198 INFO:teuthology.orchestra.run.trial169.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2026-02-25T08:41:29.198 INFO:teuthology.orchestra.run.trial169.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2026-02-25T08:41:29.198 INFO:teuthology.orchestra.run.trial169.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2026-02-25T08:41:29.198 INFO:teuthology.orchestra.run.trial169.stdout: "cephsqlite_blocklist_dead_locker": "true", 2026-02-25T08:41:29.198 INFO:teuthology.orchestra.run.trial169.stdout: "cephsqlite_lock_renewal_interval": "2000", 2026-02-25T08:41:29.198 INFO:teuthology.orchestra.run.trial169.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2026-02-25T08:41:29.198 INFO:teuthology.orchestra.run.trial169.stdout: "cephx_cluster_require_signatures": "false", 2026-02-25T08:41:29.198 INFO:teuthology.orchestra.run.trial169.stdout: "cephx_cluster_require_version": "2", 2026-02-25T08:41:29.198 INFO:teuthology.orchestra.run.trial169.stdout: "cephx_require_signatures": "false", 2026-02-25T08:41:29.198 INFO:teuthology.orchestra.run.trial169.stdout: "cephx_require_version": "2", 2026-02-25T08:41:29.198 INFO:teuthology.orchestra.run.trial169.stdout: "cephx_service_require_signatures": "false", 2026-02-25T08:41:29.198 INFO:teuthology.orchestra.run.trial169.stdout: "cephx_service_require_version": "2", 2026-02-25T08:41:29.198 INFO:teuthology.orchestra.run.trial169.stdout: "cephx_sign_messages": "true", 2026-02-25T08:41:29.199 INFO:teuthology.orchestra.run.trial169.stdout: "chdir": "", 2026-02-25T08:41:29.199 INFO:teuthology.orchestra.run.trial169.stdout: "client_acl_type": "", 2026-02-25T08:41:29.199 INFO:teuthology.orchestra.run.trial169.stdout: "client_asio_thread_count": "2", 2026-02-25T08:41:29.199 INFO:teuthology.orchestra.run.trial169.stdout: "client_cache_mid": "0.750000", 2026-02-25T08:41:29.199 INFO:teuthology.orchestra.run.trial169.stdout: "client_cache_size": "16384", 2026-02-25T08:41:29.199 INFO:teuthology.orchestra.run.trial169.stdout: "client_caps_release_delay": "5", 2026-02-25T08:41:29.199 INFO:teuthology.orchestra.run.trial169.stdout: "client_check_pool_perm": "true", 2026-02-25T08:41:29.199 INFO:teuthology.orchestra.run.trial169.stdout: "client_collect_and_send_global_metrics": "false", 2026-02-25T08:41:29.199 INFO:teuthology.orchestra.run.trial169.stdout: "client_debug_force_sync_read": "false", 2026-02-25T08:41:29.199 INFO:teuthology.orchestra.run.trial169.stdout: "client_debug_getattr_caps": "false", 2026-02-25T08:41:29.199 INFO:teuthology.orchestra.run.trial169.stdout: "client_debug_inject_features": "", 2026-02-25T08:41:29.199 INFO:teuthology.orchestra.run.trial169.stdout: "client_debug_inject_tick_delay": "0", 2026-02-25T08:41:29.199 INFO:teuthology.orchestra.run.trial169.stdout: "client_die_on_failed_dentry_invalidate": "true", 2026-02-25T08:41:29.200 INFO:teuthology.orchestra.run.trial169.stdout: "client_die_on_failed_remount": "false", 2026-02-25T08:41:29.200 INFO:teuthology.orchestra.run.trial169.stdout: "client_dirsize_rbytes": "true", 2026-02-25T08:41:29.200 INFO:teuthology.orchestra.run.trial169.stdout: "client_file_blockdiff_max_concurrent_object_scans": "16", 2026-02-25T08:41:29.200 INFO:teuthology.orchestra.run.trial169.stdout: "client_force_lazyio": "false", 2026-02-25T08:41:29.200 INFO:teuthology.orchestra.run.trial169.stdout: "client_fs": "", 2026-02-25T08:41:29.200 INFO:teuthology.orchestra.run.trial169.stdout: "client_fscrypt_as": "true", 2026-02-25T08:41:29.200 INFO:teuthology.orchestra.run.trial169.stdout: "client_fscrypt_dummy_encryption": "false", 2026-02-25T08:41:29.200 INFO:teuthology.orchestra.run.trial169.stdout: "client_inject_fixed_oldest_tid": "false", 2026-02-25T08:41:29.200 INFO:teuthology.orchestra.run.trial169.stdout: "client_inject_release_failure": "false", 2026-02-25T08:41:29.200 INFO:teuthology.orchestra.run.trial169.stdout: "client_inject_write_delay_secs": "0", 2026-02-25T08:41:29.200 INFO:teuthology.orchestra.run.trial169.stdout: "client_max_inline_size": "4096", 2026-02-25T08:41:29.200 INFO:teuthology.orchestra.run.trial169.stdout: "client_max_retries_on_remount_failure": "5", 2026-02-25T08:41:29.200 INFO:teuthology.orchestra.run.trial169.stdout: "client_mds_namespace": "", 2026-02-25T08:41:29.200 INFO:teuthology.orchestra.run.trial169.stdout: "client_metadata": "", 2026-02-25T08:41:29.201 INFO:teuthology.orchestra.run.trial169.stdout: "client_mount_gid": "-1", 2026-02-25T08:41:29.201 INFO:teuthology.orchestra.run.trial169.stdout: "client_mount_timeout": "300", 2026-02-25T08:41:29.201 INFO:teuthology.orchestra.run.trial169.stdout: "client_mount_uid": "-1", 2026-02-25T08:41:29.201 INFO:teuthology.orchestra.run.trial169.stdout: "client_mountpoint": "/", 2026-02-25T08:41:29.201 INFO:teuthology.orchestra.run.trial169.stdout: "client_notify_timeout": "10", 2026-02-25T08:41:29.201 INFO:teuthology.orchestra.run.trial169.stdout: "client_oc": "true", 2026-02-25T08:41:29.201 INFO:teuthology.orchestra.run.trial169.stdout: "client_oc_max_dirty": "104857600", 2026-02-25T08:41:29.201 INFO:teuthology.orchestra.run.trial169.stdout: "client_oc_max_dirty_age": "5.000000", 2026-02-25T08:41:29.201 INFO:teuthology.orchestra.run.trial169.stdout: "client_oc_max_objects": "1000", 2026-02-25T08:41:29.201 INFO:teuthology.orchestra.run.trial169.stdout: "client_oc_size": "209715200", 2026-02-25T08:41:29.201 INFO:teuthology.orchestra.run.trial169.stdout: "client_oc_target_dirty": "8388608", 2026-02-25T08:41:29.201 INFO:teuthology.orchestra.run.trial169.stdout: "client_permissions": "true", 2026-02-25T08:41:29.201 INFO:teuthology.orchestra.run.trial169.stdout: "client_quota": "true", 2026-02-25T08:41:29.202 INFO:teuthology.orchestra.run.trial169.stdout: "client_quota_df": "true", 2026-02-25T08:41:29.202 INFO:teuthology.orchestra.run.trial169.stdout: "client_readahead_max_bytes": "0", 2026-02-25T08:41:29.202 INFO:teuthology.orchestra.run.trial169.stdout: "client_readahead_max_periods": "4", 2026-02-25T08:41:29.202 INFO:teuthology.orchestra.run.trial169.stdout: "client_readahead_min": "131072", 2026-02-25T08:41:29.202 INFO:teuthology.orchestra.run.trial169.stdout: "client_reconnect_stale": "false", 2026-02-25T08:41:29.202 INFO:teuthology.orchestra.run.trial169.stdout: "client_respect_subvolume_snapshot_visibility": "false", 2026-02-25T08:41:29.202 INFO:teuthology.orchestra.run.trial169.stdout: "client_shutdown_timeout": "30", 2026-02-25T08:41:29.202 INFO:teuthology.orchestra.run.trial169.stdout: "client_snapdir": ".snap", 2026-02-25T08:41:29.202 INFO:teuthology.orchestra.run.trial169.stdout: "client_tick_interval": "1", 2026-02-25T08:41:29.202 INFO:teuthology.orchestra.run.trial169.stdout: "client_trace": "", 2026-02-25T08:41:29.202 INFO:teuthology.orchestra.run.trial169.stdout: "client_try_dentry_invalidate": "false", 2026-02-25T08:41:29.202 INFO:teuthology.orchestra.run.trial169.stdout: "client_use_faked_inos": "false", 2026-02-25T08:41:29.202 INFO:teuthology.orchestra.run.trial169.stdout: "client_use_random_mds": "false", 2026-02-25T08:41:29.203 INFO:teuthology.orchestra.run.trial169.stdout: "clog_to_graylog": "false", 2026-02-25T08:41:29.203 INFO:teuthology.orchestra.run.trial169.stdout: "clog_to_graylog_host": "127.0.0.1", 2026-02-25T08:41:29.203 INFO:teuthology.orchestra.run.trial169.stdout: "clog_to_graylog_port": "12201", 2026-02-25T08:41:29.203 INFO:teuthology.orchestra.run.trial169.stdout: "clog_to_monitors": "default=true", 2026-02-25T08:41:29.203 INFO:teuthology.orchestra.run.trial169.stdout: "clog_to_syslog": "false", 2026-02-25T08:41:29.203 INFO:teuthology.orchestra.run.trial169.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2026-02-25T08:41:29.203 INFO:teuthology.orchestra.run.trial169.stdout: "clog_to_syslog_level": "info", 2026-02-25T08:41:29.203 INFO:teuthology.orchestra.run.trial169.stdout: "cluster_addr": "-", 2026-02-25T08:41:29.203 INFO:teuthology.orchestra.run.trial169.stdout: "cluster_network": "", 2026-02-25T08:41:29.203 INFO:teuthology.orchestra.run.trial169.stdout: "cluster_network_interface": "", 2026-02-25T08:41:29.203 INFO:teuthology.orchestra.run.trial169.stdout: "compressor_zlib_isal": "false", 2026-02-25T08:41:29.203 INFO:teuthology.orchestra.run.trial169.stdout: "compressor_zlib_level": "5", 2026-02-25T08:41:29.203 INFO:teuthology.orchestra.run.trial169.stdout: "compressor_zlib_winsize": "-15", 2026-02-25T08:41:29.203 INFO:teuthology.orchestra.run.trial169.stdout: "compressor_zstd_level": "1", 2026-02-25T08:41:29.204 INFO:teuthology.orchestra.run.trial169.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2026-02-25T08:41:29.204 INFO:teuthology.orchestra.run.trial169.stdout: "crash_dir": "/var/lib/ceph/crash", 2026-02-25T08:41:29.204 INFO:teuthology.orchestra.run.trial169.stdout: "crimson_allow_pg_split": "true", 2026-02-25T08:41:29.204 INFO:teuthology.orchestra.run.trial169.stdout: "crimson_bluestore_cpu_set": "", 2026-02-25T08:41:29.204 INFO:teuthology.orchestra.run.trial169.stdout: "crimson_bluestore_num_threads": "6", 2026-02-25T08:41:29.204 INFO:teuthology.orchestra.run.trial169.stdout: "crimson_cpu_num": "0", 2026-02-25T08:41:29.204 INFO:teuthology.orchestra.run.trial169.stdout: "crimson_cpu_set": "", 2026-02-25T08:41:29.204 INFO:teuthology.orchestra.run.trial169.stdout: "crimson_osd_obc_lru_size": "512", 2026-02-25T08:41:29.204 INFO:teuthology.orchestra.run.trial169.stdout: "crimson_osd_scheduler_concurrency": "0", 2026-02-25T08:41:29.204 INFO:teuthology.orchestra.run.trial169.stdout: "crimson_osd_stat_interval": "0", 2026-02-25T08:41:29.204 INFO:teuthology.orchestra.run.trial169.stdout: "crimson_poll_mode": "false", 2026-02-25T08:41:29.204 INFO:teuthology.orchestra.run.trial169.stdout: "crimson_reactor_idle_poll_time_us": "200", 2026-02-25T08:41:29.204 INFO:teuthology.orchestra.run.trial169.stdout: "crimson_reactor_io_latency_goal_ms": "0.000000", 2026-02-25T08:41:29.205 INFO:teuthology.orchestra.run.trial169.stdout: "crimson_reactor_task_quota_ms": "0.500000", 2026-02-25T08:41:29.205 INFO:teuthology.orchestra.run.trial169.stdout: "crush_location": "", 2026-02-25T08:41:29.205 INFO:teuthology.orchestra.run.trial169.stdout: "crush_location_hook": "", 2026-02-25T08:41:29.205 INFO:teuthology.orchestra.run.trial169.stdout: "crush_location_hook_timeout": "10", 2026-02-25T08:41:29.205 INFO:teuthology.orchestra.run.trial169.stdout: "d4n_writecache_enabled": "false", 2026-02-25T08:41:29.205 INFO:teuthology.orchestra.run.trial169.stdout: "daemonize": "false", 2026-02-25T08:41:29.205 INFO:teuthology.orchestra.run.trial169.stdout: "daos_pool": "tank", 2026-02-25T08:41:29.205 INFO:teuthology.orchestra.run.trial169.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2026-02-25T08:41:29.205 INFO:teuthology.orchestra.run.trial169.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2026-02-25T08:41:29.205 INFO:teuthology.orchestra.run.trial169.stdout: "dbstore_db_name_prefix": "dbstore", 2026-02-25T08:41:29.205 INFO:teuthology.orchestra.run.trial169.stdout: "debug_alienstore": "0/5", 2026-02-25T08:41:29.205 INFO:teuthology.orchestra.run.trial169.stdout: "debug_allow_any_pool_priority": "false", 2026-02-25T08:41:29.205 INFO:teuthology.orchestra.run.trial169.stdout: "debug_asok": "1/5", 2026-02-25T08:41:29.206 INFO:teuthology.orchestra.run.trial169.stdout: "debug_asok_assert_abort": "false", 2026-02-25T08:41:29.206 INFO:teuthology.orchestra.run.trial169.stdout: "debug_asserts_on_shutdown": "true", 2026-02-25T08:41:29.206 INFO:teuthology.orchestra.run.trial169.stdout: "debug_auth": "20/20", 2026-02-25T08:41:29.206 INFO:teuthology.orchestra.run.trial169.stdout: "debug_bdev": "1/3", 2026-02-25T08:41:29.206 INFO:teuthology.orchestra.run.trial169.stdout: "debug_bluefs": "1/5", 2026-02-25T08:41:29.206 INFO:teuthology.orchestra.run.trial169.stdout: "debug_bluestore": "1/5", 2026-02-25T08:41:29.206 INFO:teuthology.orchestra.run.trial169.stdout: "debug_bluestore_compression": "1/5", 2026-02-25T08:41:29.206 INFO:teuthology.orchestra.run.trial169.stdout: "debug_buffer": "0/1", 2026-02-25T08:41:29.206 INFO:teuthology.orchestra.run.trial169.stdout: "debug_ceph_dedup": "0/5", 2026-02-25T08:41:29.206 INFO:teuthology.orchestra.run.trial169.stdout: "debug_ceph_exporter": "1/5", 2026-02-25T08:41:29.206 INFO:teuthology.orchestra.run.trial169.stdout: "debug_cephfs_mirror": "0/5", 2026-02-25T08:41:29.206 INFO:teuthology.orchestra.run.trial169.stdout: "debug_cephsqlite": "0/5", 2026-02-25T08:41:29.206 INFO:teuthology.orchestra.run.trial169.stdout: "debug_client": "0/5", 2026-02-25T08:41:29.207 INFO:teuthology.orchestra.run.trial169.stdout: "debug_compressor": "1/5", 2026-02-25T08:41:29.207 INFO:teuthology.orchestra.run.trial169.stdout: "debug_context": "0/1", 2026-02-25T08:41:29.207 INFO:teuthology.orchestra.run.trial169.stdout: "debug_crimson_interrupt": "0/5", 2026-02-25T08:41:29.207 INFO:teuthology.orchestra.run.trial169.stdout: "debug_crush": "1/1", 2026-02-25T08:41:29.207 INFO:teuthology.orchestra.run.trial169.stdout: "debug_crypto": "1/5", 2026-02-25T08:41:29.207 INFO:teuthology.orchestra.run.trial169.stdout: "debug_cyanstore": "0/5", 2026-02-25T08:41:29.207 INFO:teuthology.orchestra.run.trial169.stdout: "debug_deliberately_leak_memory": "false", 2026-02-25T08:41:29.207 INFO:teuthology.orchestra.run.trial169.stdout: "debug_disable_randomized_ping": "false", 2026-02-25T08:41:29.207 INFO:teuthology.orchestra.run.trial169.stdout: "debug_dpdk": "1/5", 2026-02-25T08:41:29.207 INFO:teuthology.orchestra.run.trial169.stdout: "debug_eventtrace": "1/5", 2026-02-25T08:41:29.207 INFO:teuthology.orchestra.run.trial169.stdout: "debug_filer": "0/1", 2026-02-25T08:41:29.207 INFO:teuthology.orchestra.run.trial169.stdout: "debug_filestore": "1/3", 2026-02-25T08:41:29.207 INFO:teuthology.orchestra.run.trial169.stdout: "debug_finisher": "1/1", 2026-02-25T08:41:29.208 INFO:teuthology.orchestra.run.trial169.stdout: "debug_fuse": "1/5", 2026-02-25T08:41:29.208 INFO:teuthology.orchestra.run.trial169.stdout: "debug_heartbeat_testing_span": "0", 2026-02-25T08:41:29.208 INFO:teuthology.orchestra.run.trial169.stdout: "debug_heartbeatmap": "1/5", 2026-02-25T08:41:29.208 INFO:teuthology.orchestra.run.trial169.stdout: "debug_immutable_obj_cache": "0/5", 2026-02-25T08:41:29.208 INFO:teuthology.orchestra.run.trial169.stdout: "debug_javaclient": "1/5", 2026-02-25T08:41:29.208 INFO:teuthology.orchestra.run.trial169.stdout: "debug_journal": "1/3", 2026-02-25T08:41:29.208 INFO:teuthology.orchestra.run.trial169.stdout: "debug_journaler": "0/5", 2026-02-25T08:41:29.208 INFO:teuthology.orchestra.run.trial169.stdout: "debug_kstore": "1/5", 2026-02-25T08:41:29.208 INFO:teuthology.orchestra.run.trial169.stdout: "debug_lockdep": "0/1", 2026-02-25T08:41:29.208 INFO:teuthology.orchestra.run.trial169.stdout: "debug_mclock": "1/5", 2026-02-25T08:41:29.208 INFO:teuthology.orchestra.run.trial169.stdout: "debug_mds": "1/5", 2026-02-25T08:41:29.208 INFO:teuthology.orchestra.run.trial169.stdout: "debug_mds_balancer": "1/5", 2026-02-25T08:41:29.208 INFO:teuthology.orchestra.run.trial169.stdout: "debug_mds_locker": "1/5", 2026-02-25T08:41:29.209 INFO:teuthology.orchestra.run.trial169.stdout: "debug_mds_log": "1/5", 2026-02-25T08:41:29.209 INFO:teuthology.orchestra.run.trial169.stdout: "debug_mds_log_expire": "1/5", 2026-02-25T08:41:29.209 INFO:teuthology.orchestra.run.trial169.stdout: "debug_mds_migrator": "1/5", 2026-02-25T08:41:29.209 INFO:teuthology.orchestra.run.trial169.stdout: "debug_mds_quiesce": "3/5", 2026-02-25T08:41:29.209 INFO:teuthology.orchestra.run.trial169.stdout: "debug_memstore": "1/5", 2026-02-25T08:41:29.209 INFO:teuthology.orchestra.run.trial169.stdout: "debug_mgr": "2/5", 2026-02-25T08:41:29.209 INFO:teuthology.orchestra.run.trial169.stdout: "debug_mgrc": "1/5", 2026-02-25T08:41:29.209 INFO:teuthology.orchestra.run.trial169.stdout: "debug_mon": "20/20", 2026-02-25T08:41:29.209 INFO:teuthology.orchestra.run.trial169.stdout: "debug_monc": "0/10", 2026-02-25T08:41:29.209 INFO:teuthology.orchestra.run.trial169.stdout: "debug_ms": "1/1", 2026-02-25T08:41:29.209 INFO:teuthology.orchestra.run.trial169.stdout: "debug_none": "0/5", 2026-02-25T08:41:29.209 INFO:teuthology.orchestra.run.trial169.stdout: "debug_objclass": "0/5", 2026-02-25T08:41:29.209 INFO:teuthology.orchestra.run.trial169.stdout: "debug_objectcacher": "0/5", 2026-02-25T08:41:29.209 INFO:teuthology.orchestra.run.trial169.stdout: "debug_objecter": "0/1", 2026-02-25T08:41:29.210 INFO:teuthology.orchestra.run.trial169.stdout: "debug_optracker": "0/5", 2026-02-25T08:41:29.210 INFO:teuthology.orchestra.run.trial169.stdout: "debug_osd": "1/5", 2026-02-25T08:41:29.210 INFO:teuthology.orchestra.run.trial169.stdout: "debug_paxos": "20/20", 2026-02-25T08:41:29.210 INFO:teuthology.orchestra.run.trial169.stdout: "debug_perfcounter": "1/5", 2026-02-25T08:41:29.210 INFO:teuthology.orchestra.run.trial169.stdout: "debug_prioritycache": "1/5", 2026-02-25T08:41:29.210 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rados": "0/5", 2026-02-25T08:41:29.210 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rbd": "0/5", 2026-02-25T08:41:29.210 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rbd_mirror": "0/5", 2026-02-25T08:41:29.210 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rbd_pwl": "0/5", 2026-02-25T08:41:29.210 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rbd_replay": "0/5", 2026-02-25T08:41:29.210 INFO:teuthology.orchestra.run.trial169.stdout: "debug_refs": "0/0", 2026-02-25T08:41:29.210 INFO:teuthology.orchestra.run.trial169.stdout: "debug_reserver": "1/1", 2026-02-25T08:41:29.210 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rgw": "1/5", 2026-02-25T08:41:29.211 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rgw_access": "1/5", 2026-02-25T08:41:29.211 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rgw_bucket_logging": "1/5", 2026-02-25T08:41:29.211 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rgw_datacache": "1/5", 2026-02-25T08:41:29.211 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rgw_dbstore": "1/5", 2026-02-25T08:41:29.211 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rgw_dedup": "1/5", 2026-02-25T08:41:29.211 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rgw_flight": "1/5", 2026-02-25T08:41:29.211 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rgw_lifecycle": "1/5", 2026-02-25T08:41:29.211 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rgw_notification": "1/5", 2026-02-25T08:41:29.211 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rgw_restore": "1/5", 2026-02-25T08:41:29.211 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rgw_sync": "1/5", 2026-02-25T08:41:29.211 INFO:teuthology.orchestra.run.trial169.stdout: "debug_rocksdb": "4/5", 2026-02-25T08:41:29.211 INFO:teuthology.orchestra.run.trial169.stdout: "debug_seastore": "0/5", 2026-02-25T08:41:29.211 INFO:teuthology.orchestra.run.trial169.stdout: "debug_seastore_backref": "0/5", 2026-02-25T08:41:29.211 INFO:teuthology.orchestra.run.trial169.stdout: "debug_seastore_cache": "0/5", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_seastore_cleaner": "0/5", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_seastore_device": "0/5", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_seastore_epm": "0/5", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_seastore_fixedkv_tree": "0/5", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_seastore_journal": "0/5", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_seastore_lba": "0/5", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_seastore_odata": "0/5", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_seastore_omap": "0/5", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_seastore_onode": "0/5", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_seastore_t": "0/5", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_seastore_tm": "0/5", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_striper": "0/1", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_test": "0/5", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_throttle": "1/1", 2026-02-25T08:41:29.212 INFO:teuthology.orchestra.run.trial169.stdout: "debug_timer": "0/1", 2026-02-25T08:41:29.213 INFO:teuthology.orchestra.run.trial169.stdout: "debug_tp": "0/5", 2026-02-25T08:41:29.213 INFO:teuthology.orchestra.run.trial169.stdout: "debug_trace": "1/5", 2026-02-25T08:41:29.213 INFO:teuthology.orchestra.run.trial169.stdout: "defer_client_eviction_on_laggy_osds": "false", 2026-02-25T08:41:29.213 INFO:teuthology.orchestra.run.trial169.stdout: "device_failure_prediction_mode": "none", 2026-02-25T08:41:29.213 INFO:teuthology.orchestra.run.trial169.stdout: "ec_extent_cache_size": "10485760", 2026-02-25T08:41:29.213 INFO:teuthology.orchestra.run.trial169.stdout: "ec_pdw_write_mode": "0", 2026-02-25T08:41:29.213 INFO:teuthology.orchestra.run.trial169.stdout: "enable_availability_tracking": "false", 2026-02-25T08:41:29.213 INFO:teuthology.orchestra.run.trial169.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "crimson", 2026-02-25T08:41:29.213 INFO:teuthology.orchestra.run.trial169.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2026-02-25T08:41:29.213 INFO:teuthology.orchestra.run.trial169.stdout: "err_to_graylog": "false", 2026-02-25T08:41:29.213 INFO:teuthology.orchestra.run.trial169.stdout: "err_to_journald": "false", 2026-02-25T08:41:29.213 INFO:teuthology.orchestra.run.trial169.stdout: "err_to_stderr": "true", 2026-02-25T08:41:29.213 INFO:teuthology.orchestra.run.trial169.stdout: "err_to_syslog": "false", 2026-02-25T08:41:29.213 INFO:teuthology.orchestra.run.trial169.stdout: "event_tracing": "false", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "exporter_addr": "0.0.0.0", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "exporter_cert_file": "", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "exporter_http_port": "9926", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "exporter_key_file": "", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "exporter_prio_limit": "5", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "exporter_sock_dir": "/var/run/ceph/", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "exporter_sort_metrics": "true", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "exporter_stats_period": "5", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "fake_statfs_for_testing": "0", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "fatal_signal_handlers": "true", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "filer_max_purge_ops": "10", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "filer_max_truncate_ops": "128", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_apply_finisher_threads": "1", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_blackhole": "false", 2026-02-25T08:41:29.214 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_btrfs_clone_range": "true", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_btrfs_snap": "true", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_caller_concurrency": "10", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_collect_device_partition_information": "true", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_commit_timeout": "600.000000", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_debug_inject_read_err": "false", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_debug_omap_check": "false", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_debug_verify_split": "false", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_dump_file": "", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_expected_throughput_ops": "200.000000", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_fadvise": "true", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_fail_eio": "true", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_fd_cache_shards": "16", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_fd_cache_size": "128", 2026-02-25T08:41:29.215 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_fiemap": "false", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_fiemap_threshold": "4096", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_fsync_flushes_journal_data": "false", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_index_retry_probability": "0.000000", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_inject_stall": "0", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_journal_parallel": "false", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_journal_trailing": "false", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_journal_writeahead": "false", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_kill_at": "0", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_max_alloc_hint_size": "1048576", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_max_inline_xattr_size": "0", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_max_inline_xattr_size_other": "512", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_max_inline_xattrs": "0", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_max_inline_xattrs_other": "2", 2026-02-25T08:41:29.216 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_max_inline_xattrs_xfs": "10", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_max_sync_interval": "5.000000", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_max_xattr_value_size": "0", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_max_xattr_value_size_other": "1024", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_merge_threshold": "-10", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_min_sync_interval": "0.010000", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_odsync_write": "false", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_omap_backend": "rocksdb", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_omap_backend_path": "", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_omap_header_cache_size": "1024", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_ondisk_finisher_threads": "1", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_op_thread_suicide_timeout": "180", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_op_thread_timeout": "60", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_op_threads": "2", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_punch_hole": "false", 2026-02-25T08:41:29.217 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_queue_high_threshhold": "0.900000", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_queue_low_threshhold": "0.300000", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_queue_max_bytes": "104857600", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_queue_max_ops": "50", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_seek_data_hole": "false", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_sloppy_crc": "false", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_sloppy_crc_block_size": "65536", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_splice": "false", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_split_multiple": "2", 2026-02-25T08:41:29.218 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_split_rand_factor": "20", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_update_to": "1000", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_wbthrottle_enable": "true", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_xfs_extsize": "false", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "filestore_zfs_snap": "false", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "fio_dir": "/tmp/fio", 2026-02-25T08:41:29.219 INFO:teuthology.orchestra.run.trial169.stdout: "fsid": "a3da792f-4239-4b1a-ad7d-525e6c05ee6a", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_allow_other": "true", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_atomic_o_trunc": "true", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_big_writes": "true", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_debug": "false", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_default_permissions": "false", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_disable_pagecache": "false", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_max_write": "0", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_multithreaded": "true", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_require_active_mds": "true", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_set_user_groups": "true", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_splice_move": "true", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_splice_read": "true", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_splice_write": "true", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_syncfs_on_mksnap": "true", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "fuse_use_invalidate_cb": "true", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "gss_target_name": "ceph", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "heartbeat_file": "", 2026-02-25T08:41:29.220 INFO:teuthology.orchestra.run.trial169.stdout: "heartbeat_inject_failure": "0", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "heartbeat_interval": "5", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "host": "trial169", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "immutable_object_cache_max_inflight_ops": "128", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "immutable_object_cache_max_size": "1073741824", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "immutable_object_cache_qos_bps_burst": "0", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "immutable_object_cache_qos_bps_limit": "0", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "immutable_object_cache_qos_iops_burst": "0", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "immutable_object_cache_qos_iops_limit": "0", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "immutable_object_cache_watermark": "0.900000", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "inject_early_sigterm": "false", 2026-02-25T08:41:29.221 INFO:teuthology.orchestra.run.trial169.stdout: "jaeger_agent_port": "6799", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "jaeger_tracing_enable": "false", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_aio": "true", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_align_min_size": "65536", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_block_align": "true", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_block_size": "4096", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_dio": "true", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_discard": "false", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_force_aio": "false", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_ignore_corruption": "false", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_max_write_bytes": "10485760", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_max_write_entries": "100", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_replay_from": "0", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_throttle_high_multiple": "0.000000", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_throttle_high_threshhold": "0.900000", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_throttle_low_threshhold": "0.600000", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_throttle_max_multiple": "0.000000", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_write_header_frequency": "0", 2026-02-25T08:41:29.222 INFO:teuthology.orchestra.run.trial169.stdout: "journal_zero_on_create": "false", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "journaler_prefetch_periods": "10", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "journaler_prezero_periods": "5", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "journaler_write_head_interval": "15", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "key": "", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "keyfile": "", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "librados_thread_count": "2", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "lockdep": "false", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "lockdep_force_backtrace": "false", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "log_coarse_timestamps": "true", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "log_flush_on_exit": "false", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "log_graylog_host": "127.0.0.1", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "log_graylog_port": "12201", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "log_max_new": "1000", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "log_max_recent": "10000", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "log_stderr_prefix": "", 2026-02-25T08:41:29.223 INFO:teuthology.orchestra.run.trial169.stdout: "log_stop_at_utilization": "0.970000", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "log_to_file": "true", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "log_to_graylog": "false", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "log_to_journald": "false", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "log_to_stderr": "false", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "log_to_syslog": "false", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "max_rotating_auth_attempts": "10", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "mds_action_on_write_error": "1", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "mds_allow_async_dirops": "true", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "mds_allow_batched_ops": "true", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "mds_alternate_name_max": "8192", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "mds_asio_thread_count": "2", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_export_pin": "true", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_fragment_dirs": "true", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_fragment_interval": "5", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_fragment_size_max": "100000", 2026-02-25T08:41:29.224 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_idle_threshold": "0.000000", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_interval": "10", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_max": "-1", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_max_until": "-1", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_merge_size": "50", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_midchunk": "0.300000", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_min_rebalance": "0.100000", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_min_start": "0.200000", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_minchunk": "0.001000", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_mode": "0", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_need_max": "1.200000", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_need_min": "0.800000", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_overload_epochs": "2", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_replicate_threshold": "8000.000000", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_sample_interval": "3.000000", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_split_bits": "3", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_split_rd": "25000.000000", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_split_size": "10000", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_split_wr": "10000.000000", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_target_decay": "10.000000", 2026-02-25T08:41:29.225 INFO:teuthology.orchestra.run.trial169.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_beacon_grace": "15.000000", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_beacon_interval": "4.000000", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_beacon_mon_down_grace": "60", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_cache_memory_limit": "4294967296", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_cache_mid": "0.700000", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_cache_quiesce_delay": "0", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_cache_quiesce_sleep": "200", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_cache_quiesce_splitauth": "true", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_cache_quiesce_threshold": "524288", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_cache_release_free_interval": "10", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_cache_reservation": "0.050000", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_cache_trim_decay_rate": "1.000000", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_cache_trim_interval": "1", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_cache_trim_threshold": "262144", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_client_delegate_inos_pct": "50", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_client_prealloc_inos": "1000", 2026-02-25T08:41:29.226 INFO:teuthology.orchestra.run.trial169.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_connect_bootstrapping": "false", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_damage_table_max_entries": "10000", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_debug_auth_pins": "false", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_debug_frag": "false", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_debug_scatterstat": "false", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_debug_subtrees": "false", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_decay_halflife": "5.000000", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_default_dir_hash": "2", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_defer_session_stale": "true", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_delay_journal_replay_for_testing": "0", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_deny_all_reconnect": "false", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_dir_keys_per_op": "16384", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_dir_max_commit_size": "10", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_dir_max_entries": "0", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_dir_prefetch": "true", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_dirstat_min_interval": "1.000000", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_dump_cache_after_rejoin": "false", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_dump_cache_on_map": "false", 2026-02-25T08:41:29.227 INFO:teuthology.orchestra.run.trial169.stdout: "mds_dump_cache_threshold_file": "0", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_early_reply": "true", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_enable_op_tracker": "true", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_enforce_unique_name": "true", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_export_ephemeral_distributed": "true", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_export_ephemeral_random": "true", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_export_ephemeral_random_max": "0.010000", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_extraordinary_events_dump_interval": "0", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_file_blockdiff_max_concurrent_object_scans": "16", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_forward_all_requests_to_auth": "false", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_freeze_tree_timeout": "30.000000", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_fscrypt_last_block_max_size": "4096", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_go_bad_corrupt_dentry": "true", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_health_cache_threshold": "1.500000", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_health_summarize_threshold": "10", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_heartbeat_grace": "15.000000", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_heartbeat_reset_grace": "1000", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_inject_health_dummy": "false", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2026-02-25T08:41:29.228 INFO:teuthology.orchestra.run.trial169.stdout: "mds_inject_migrator_session_race": "false", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_inject_skip_replaying_inotable": "false", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_join_fs": "", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_journal_format": "1", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_kill_after_journal_logs_flushed": "false", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_kill_create_at": "0", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_kill_dirfrag_at": "0", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_kill_export_at": "0", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_kill_import_at": "0", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_kill_journal_at": "0", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_kill_journal_expire_at": "0", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_kill_journal_replay_at": "0", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_kill_link_at": "0", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_kill_mdstable_at": "0", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_kill_openc_at": "0", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_kill_rename_at": "0", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_kill_shutdown_at": "0", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_log_event_large_threshold": "524288", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_log_events_per_segment": "1024", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_log_max_events": "-1", 2026-02-25T08:41:29.229 INFO:teuthology.orchestra.run.trial169.stdout: "mds_log_max_segments": "128", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_log_minor_segments_per_major_segment": "16", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_log_pause": "false", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_log_segment_size": "0", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_log_skip_corrupt_events": "false", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_log_skip_unbounded_events": "false", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_log_trim_decay_rate": "1.000000", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_log_trim_threshold": "128", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_log_trim_upkeep_interval": "1000", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_log_warn_factor": "2.000000", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_max_caps_per_client": "1048576", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_max_completed_flushes": "100000", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_max_completed_requests": "100000", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_max_export_size": "20971520", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_max_file_recover": "32", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_max_purge_files": "64", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_max_purge_ops": "8192", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_max_scrub_ops_in_progress": "5", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_max_snaps_per_dir": "100", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_metrics_update_interval": "2", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_min_caps_per_client": "100", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_min_caps_working_set": "10000", 2026-02-25T08:41:29.230 INFO:teuthology.orchestra.run.trial169.stdout: "mds_mon_shutdown_timeout": "5.000000", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_numa_node": "-1", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_oft_prefetch_dirfrags": "false", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_op_complaint_time": "30.000000", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_op_history_duration": "600", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_op_history_size": "20", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_op_history_slow_op_size": "20", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_op_history_slow_op_threshold": "10", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_op_log_threshold": "5", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_ping_grace": "15", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_ping_interval": "5", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_recall_global_max_decay_threshold": "131072", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_recall_max_caps": "30000", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_recall_max_decay_rate": "1.500000", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_recall_max_decay_threshold": "131072", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_recall_warning_decay_rate": "60.000000", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_recall_warning_threshold": "262144", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_reconnect_timeout": "45.000000", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_replay_interval": "1.000000", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_replay_unsafe_with_closed_session": "false", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_request_load_average_decay_rate": "60.000000", 2026-02-25T08:41:29.231 INFO:teuthology.orchestra.run.trial169.stdout: "mds_root_ino_gid": "0", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_root_ino_uid": "0", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_scatter_nudge_interval": "5.000000", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_scrub_stats_review_period": "1", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_server_dispatch_client_request_delay": "0", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_session_blocklist_on_evict": "true", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_session_blocklist_on_timeout": "true", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_session_cache_liveness_magnitude": "10", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_session_cap_acquisition_throttle": "100000", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_session_metadata_threshold": "16777216", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_sessionmap_keys_per_op": "1024", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_shutdown_check": "0", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_skip_ino": "0", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_sleep_rank_change": "0.000000", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_snap_max_uid": "4294967294", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_snap_min_uid": "0", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_snap_rstat": "false", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_standby_replay_damaged": "false", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_symlink_recovery": "true", 2026-02-25T08:41:29.232 INFO:teuthology.orchestra.run.trial169.stdout: "mds_task_status_update_interval": "2.000000", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mds_thrash_exports": "0", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mds_thrash_fragments": "0", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mds_tick_interval": "5.000000", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mds_use_global_snaprealm_seq_for_subvol": "true", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mds_valgrind_exit": "false", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mds_verify_backtrace": "1", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mds_verify_scatter": "false", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mds_wipe_ino_prealloc": "false", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mds_wipe_sessions": "false", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mempool_debug": "false", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "memstore_debug_omit_block_device_write": "false", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "memstore_device_bytes": "1073741824", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "memstore_page_set": "false", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "memstore_page_size": "65536", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_client_bytes": "134217728", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_client_messages": "512", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_connect_retry_interval": "1.000000", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_disabled_modules": "", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_enable_op_tracker": "true", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_initial_modules": "iostat nfs", 2026-02-25T08:41:29.233 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_max_pg_creating": "1024", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_max_pg_num_change": "128", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_mds_bytes": "134217728", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_mds_messages": "128", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_mon_bytes": "134217728", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_mon_messages": "128", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_num_op_tracker_shard": "32", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_op_complaint_time": "30.000000", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_op_history_duration": "600", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_op_history_size": "20", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_op_history_slow_op_size": "20", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_op_log_threshold": "5", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_osd_bytes": "536870912", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_osd_messages": "8192", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_pool": "true", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_service_beacon_grace": "60.000000", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_standby_modules": "true", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_stats_period": "5", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_stats_threshold": "5", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_tick_period": "2", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mgr_ttl_cache_expire_seconds": "0", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mon_accept_timeout_factor": "2.000000", 2026-02-25T08:41:29.234 INFO:teuthology.orchestra.run.trial169.stdout: "mon_allow_pool_delete": "true", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_allow_pool_size_one": "true", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_auth_validate_all_caps": "true", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_client_bytes": "104857600", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_client_directed_command_retry": "2", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_client_hunt_interval": "3.000000", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_client_hunt_on_resend": "true", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_client_hunt_parallel": "3", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_client_log_interval": "1.000000", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_client_max_log_entries_per_message": "1000", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_client_ping_interval": "10.000000", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_client_ping_timeout": "30.000000", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_client_target_rank": "-1", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_clock_drift_allowed": "1.000000", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_cluster_log_level": "debug", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_cluster_log_to_file": "true", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_cluster_log_to_graylog": "false", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2026-02-25T08:41:29.235 INFO:teuthology.orchestra.run.trial169.stdout: "mon_cluster_log_to_graylog_port": "12201", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_cluster_log_to_journald": "false", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_cluster_log_to_stderr": "false", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_cluster_log_to_syslog": "default=false", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_compact_on_bootstrap": "false", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_compact_on_start": "false", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_compact_on_trim": "true", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_con_tracker_persist_interval": "10", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_con_tracker_score_halflife": "43200", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_config_key_max_entry_size": "65536", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_cpu_threads": "4", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_crush_min_required_version": "hammer", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_daemon_bytes": "419430400", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_data_avail_crit": "5", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_data_avail_warn": "5", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_data_size_warn": "16106127360", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_debug_block_osdmap_trim": "false", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_debug_deprecated_as_obsolete": "false", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_debug_dump_json": "false", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_debug_dump_transactions": "false", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_debug_extra_checks": "false", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_debug_no_initial_persistent_features": "false", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2026-02-25T08:41:29.236 INFO:teuthology.orchestra.run.trial169.stdout: "mon_debug_no_require_squid": "false", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_debug_no_require_tentacle": "false", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_delta_reset_interval": "10.000000", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_dns_srv_name": "ceph-mon", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_down_added_grace": "180", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_down_mkfs_grace": "120", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_down_uptime_grace": "60", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_election_default_strategy": "1", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_election_timeout": "5.000000", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_elector_ping_divisor": "2", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_elector_ping_timeout": "2.000000", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_enable_op_tracker": "true", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_fake_pool_delete": "false", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_force_quorum_join": "false", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_fsmap_prune_threshold": "300", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_globalid_prealloc": "10000", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_health_detail_to_clog": "false", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_health_log_update_period": "5", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_health_max_detail": "50", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_health_to_clog": "true", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_health_to_clog_interval": "600", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_host": "10.20.193.169", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_host_override": "", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_initial_members": "", 2026-02-25T08:41:29.237 INFO:teuthology.orchestra.run.trial169.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_inject_transaction_delay_max": "10.000000", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_keyvaluedb": "rocksdb", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_lease": "5.000000", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_lease_renew_interval_factor": "0.600000", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_log_full_interval": "50", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_log_max": "10000", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_log_max_summary": "50", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_max_log_entries_per_event": "4096", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_max_log_epochs": "500", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_max_mdsmap_epochs": "500", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_max_mgrmap_epochs": "500", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_max_nvmeof_epochs": "500", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_max_osd": "10000", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_max_pg_per_osd": "10000", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_max_pool_pg_num": "65536", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_max_snap_prune_per_epoch": "100", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_mds_blocklist_interval": "86400.000000", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_mds_force_trim_to": "0", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_mds_skip_sanity": "false", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_memory_autotune": "true", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_memory_target": "2147483648", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_mgr_beacon_grace": "30", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_mgr_digest_period": "5", 2026-02-25T08:41:29.238 INFO:teuthology.orchestra.run.trial169.stdout: "mon_mgr_inactive_grace": "60", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_mgr_mkfs_grace": "240", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_min_osdmap_epochs": "50", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_netsplit_grace_period": "9", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_nvmeofgw_beacon_grace": "7", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_nvmeofgw_beacons_till_ack": "15", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_nvmeofgw_delete_grace": "900", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_nvmeofgw_skip_failovers_interval": "16", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_nvmeofgw_wrong_map_ignore_sec": "15", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_op_complaint_time": "30", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_op_history_duration": "600", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_op_history_size": "20", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_op_history_slow_op_size": "20", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_op_history_slow_op_threshold": "10", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_op_log_threshold": "5", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_adjust_down_out_interval": "true", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_auto_mark_in": "false", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_auto_mark_new_in": "true", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_cache_size": "500", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_cache_size_min": "134217728", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_crush_smoke_test": "true", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_destroyed_out_interval": "600", 2026-02-25T08:41:29.239 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_down_out_interval": "600", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_down_out_subtree_limit": "rack", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_force_trim_to": "0", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_full_ratio": "0.950000", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_laggy_halflife": "3600", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_laggy_max_interval": "300", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_laggy_weight": "0.300000", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_max_initial_pgs": "1024", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_min_down_reporters": "2", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_min_in_ratio": "0.750000", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_min_up_ratio": "0.300000", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_nearfull_ratio": "0.850000", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_prime_pg_temp": "true", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_report_timeout": "900", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_reporter_subtree_level": "osd", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_warn_num_repaired": "10", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osd_warn_op_age": "32.000000", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osdmap_full_prune_enabled": "true", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osdmap_full_prune_interval": "2", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osdmap_full_prune_min": "15", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_osdmap_full_prune_txsize": "2", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2026-02-25T08:41:29.240 INFO:teuthology.orchestra.run.trial169.stdout: "mon_pg_stuck_threshold": "60", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_pg_warn_min_objects": "10000", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_pg_warn_min_per_osd": "0", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_pg_warn_min_pool_objects": "1000", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_pool_quota_crit_threshold": "0", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_pool_quota_warn_threshold": "0", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_probe_timeout": "2.000000", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_reweight_max_change": "0.050000", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_reweight_max_osds": "4", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_reweight_min_bytes_per_osd": "10", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_reweight_min_pgs_per_osd": "4", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_scrub_interval": "86400", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_scrub_max_keys": "100", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_scrub_timeout": "300", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_session_timeout": "300", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_smart_report_timeout": "5", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_stat_smooth_intervals": "6", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_stretch_pool_min_size": "2", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_stretch_pool_size": "4", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_subscribe_interval": "86400.000000", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_sync_debug": "false", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_sync_max_payload_keys": "2000", 2026-02-25T08:41:29.241 INFO:teuthology.orchestra.run.trial169.stdout: "mon_sync_max_payload_size": "1048576", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_sync_provider_kill_at": "0", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_sync_requester_kill_at": "0", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_sync_timeout": "60.000000", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_target_pg_per_osd": "100", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_tick_interval": "5", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_timecheck_interval": "300.000000", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_timecheck_skew_interval": "30.000000", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_use_min_delay_socket": "false", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_older_version_delay": "604800", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_filestore_osds": "false", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_misplaced": "false", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_older_version": "true", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_pool_no_app": "true", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_pool_no_app_grace": "300", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_pool_no_redundancy": "false", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_on_too_few_osds": "false", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2026-02-25T08:41:29.242 INFO:teuthology.orchestra.run.trial169.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "monmap": "", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "motr_tracing_enabled": "false", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_op_threads": "3", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_buffer_size": "131072", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_cm": "false", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_device_name": "", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_dscp": "96", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_enable_hugepage": "false", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_gid_idx": "0", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_local_gid": "", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_polling_us": "1000", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_port_num": "1", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_receive_buffers": "32768", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_receive_queue_len": "4096", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_roce_ver": "1", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_send_buffers": "1024", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_sl": "3", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_support_srq": "true", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_rdma_type": "ib", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_async_reap_threshold": "5", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_bind_before_connect": "false", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_bind_ipv4": "true", 2026-02-25T08:41:29.243 INFO:teuthology.orchestra.run.trial169.stdout: "ms_bind_ipv6": "false", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_bind_msgr1": "true", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_bind_msgr2": "true", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_bind_port_max": "7568", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_bind_port_min": "6800", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_bind_prefer_ipv4": "false", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_bind_retry_count": "3", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_bind_retry_delay": "5", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_blackhole_client": "false", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_blackhole_mds": "false", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_blackhole_mgr": "false", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_blackhole_mon": "false", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_blackhole_osd": "false", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_client_mode": "crc", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_client_throttle_retry_time_interval": "5000", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_cluster_mode": "crc", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_cluster_type": "", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_compress_secure": "false", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_connection_idle_timeout": "900", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_connection_ready_timeout": "10", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_crc_data": "true", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_crc_header": "true", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_die_on_bad_msg": "false", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_die_on_bug": "true", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_die_on_old_message": "true", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_die_on_skipped_message": "false", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_die_on_unhandled_msg": "false", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dispatch_throttle_bytes": "104857600", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_coremask": "0xF", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_debug_allow_loopback": "false", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_devs_allowlist": "", 2026-02-25T08:41:29.244 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_enable_tso": "true", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_host_ipv4_addr": "", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_hugepages": "", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_hw_flow_control": "true", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_lro": "true", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_memory_channel": "4", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_pmd": "", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_port_id": "0", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dump_corrupt_message_level": "1", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_dump_on_send": "false", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_initial_backoff": "0.200000", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_inject_delay_max": "1.000000", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_inject_delay_probability": "0.000000", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_inject_delay_type": "", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_inject_internal_delays": "0.000000", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_inject_network_congestion": "0", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_inject_socket_failures": "0", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_learn_addr_from_peer": "true", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_max_accept_failures": "4", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_max_backoff": "15.000000", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_mon_client_mode": "crc", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_mon_cluster_mode": "crc", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_mon_service_mode": "crc", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_osd_compress_min_size": "1024", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_osd_compress_mode": "none", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_osd_compression_algorithm": "snappy", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_pq_min_cost": "65536", 2026-02-25T08:41:29.245 INFO:teuthology.orchestra.run.trial169.stdout: "ms_public_type": "", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "ms_service_mode": "crc", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "ms_tcp_listen_backlog": "512", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "ms_tcp_nodelay": "true", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "ms_tcp_prefetch_max_size": "65536", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "ms_tcp_rcvbuf": "0", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "ms_time_events_min_wait_interval": "1000", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "ms_type": "async+posix", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "no_config_file": "false", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "nvmeof_mon_client_connect_panic": "30", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "nvmeof_mon_client_tick_period": "1", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "objecter_completion_locks_per_session": "32", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "objecter_debug_inject_relock_delay": "false", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "objecter_inflight_op_bytes": "104857600", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "objecter_inflight_ops": "1024", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "objecter_inject_no_watch_ping": "false", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "objecter_retry_writes_after_first_reply": "false", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "objecter_tick_interval": "5.000000", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "objecter_timeout": "10.000000", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "objectstore_blackhole": "false", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "openssl_conf": "", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "osd_agent_delay_time": "5.000000", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "osd_agent_hist_halflife": "1000", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "osd_agent_max_low_ops": "2", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "osd_agent_max_ops": "4", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "osd_agent_min_evict_effort": "0.100000", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "osd_agent_quantize_effort": "0.100000", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "osd_agent_slop": "0.020000", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "osd_aggregated_slow_ops_logging": "true", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "osd_allow_recovery_below_min_size": "true", 2026-02-25T08:41:29.246 INFO:teuthology.orchestra.run.trial169.stdout: "osd_asio_thread_count": "2", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_async_recovery_min_cost": "100", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_auto_mark_unfound_lost": "false", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_backfill_retry_interval": "30.000000", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_backfill_scan_max": "512", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_backfill_scan_min": "64", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_backoff_on_degraded": "false", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_backoff_on_peering": "false", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_backoff_on_unfound": "true", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_beacon_report_interval": "300", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_bench_duration": "30", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_bench_large_size_max_throughput": "104857600", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_bench_max_block_size": "67108864", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_bench_small_size_max_iops": "100", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_blkin_trace_all": "false", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_blocked_scrub_grace_period": "120", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_check_for_log_corruption": "false", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_check_max_object_name_len_on_startup": "true", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo sem_set", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo sem_set", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_class_update_on_start": "true", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_client_message_cap": "256", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_client_message_size_cap": "524288000", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_client_op_priority": "63", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_client_watch_timeout": "30", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_command_max_records": "256", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_compact_on_start": "false", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_copyfrom_max_chunk": "8388608", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_crush_chooseleaf_type": "0", 2026-02-25T08:41:29.247 INFO:teuthology.orchestra.run.trial169.stdout: "osd_crush_initial_weight": "-1.000000", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_crush_update_on_start": "true", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_crush_update_weight_set": "true", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_drop_ping_duration": "0", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_drop_ping_probability": "0.000000", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_feed_pullee": "-1", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_inject_copyfrom_error": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_misdirected_ops": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_no_acting_change": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_no_purge_strays": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_op_order": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_pg_log_writeout": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_pretend_recovery_active": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_random_push_read_error": "0.000000", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_shutdown": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_trim_objects": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_verify_cached_snaps": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_verify_missing_on_start": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_verify_snaps": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_debug_verify_stray_on_activate": "false", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_deep_scrub_interval": "604800.000000", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_deep_scrub_interval_cv": "0.200000", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_deep_scrub_keys": "1024", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2026-02-25T08:41:29.248 INFO:teuthology.orchestra.run.trial169.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_deep_scrub_stride": "4194304", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_default_data_pool_replay_window": "5", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_default_notify_timeout": "30", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_delete_sleep": "0.000000", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_delete_sleep_hdd": "5.000000", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_delete_sleep_hybrid": "1.000000", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_delete_sleep_ssd": "1.000000", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_discard_disconnected_ops": "true", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_ec_partial_reads": "true", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_enable_op_tracker": "true", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_extblkdev_plugins": "vdo", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_failsafe_full_ratio": "0.970000", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_fast_fail_on_connection_refused": "true", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_fast_info": "true", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_fast_shutdown": "true", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_fast_shutdown_notify_mon": "true", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_fast_shutdown_timeout": "15", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_find_best_info_ignore_history_les": "false", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_force_auth_primary_missing_objects": "100", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_function_tracing": "false", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_heartbeat_grace": "20", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_heartbeat_interval": "6", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_heartbeat_min_peers": "10", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_heartbeat_min_size": "2000", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_heartbeat_stale": "600", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_hit_set_max_size": "100000", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_hit_set_min_size": "1000", 2026-02-25T08:41:29.249 INFO:teuthology.orchestra.run.trial169.stdout: "osd_hit_set_namespace": ".ceph-internal", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_ignore_stale_divergent_priors": "false", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_inject_failure_on_pg_removal": "false", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_journal_flush_on_shutdown": "true", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_journal_size": "5120", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_kill_backfill_at": "0", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_loop_before_reset_tphandle": "64", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_map_cache_size": "50", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_map_dedup": "true", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_map_message_max": "40", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_map_message_max_bytes": "10485760", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_map_share_max_epochs": "40", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_attr_name_len": "100", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_attr_size": "0", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_backfills": "1", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_markdown_count": "5", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_markdown_period": "600", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_object_name_len": "2048", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_object_namespace_len": "256", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_object_size": "134217728", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_omap_entries_per_request": "1024", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_pg_blocked_by": "16", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_pg_log_entries": "10000", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_pgls": "1024", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_push_cost": "8388608", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_push_objects": "10", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_scrubs": "3", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_trimming_pgs": "2", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_write_op_reply_len": "64", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_max_write_size": "90", 2026-02-25T08:41:29.250 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_iops_capacity_low_threshold_hdd": "50.000000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_iops_capacity_low_threshold_ssd": "1000.000000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_override_recovery_settings": "false", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_profile": "balanced", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_scheduler_client_wgt": "1", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mclock_skip_benchmark": "false", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_memory_base": "805306368", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_memory_cache_min": "134217728", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_memory_cache_resize_interval": "1.000000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_memory_expected_fragmentation": "0.150000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_memory_target": "4294967296", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_memory_target_autotune": "false", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_min_pg_log_entries": "250", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_min_recovery_priority": "0", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mon_heartbeat_interval": "30", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mon_report_interval": "5", 2026-02-25T08:41:29.251 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mon_report_max_in_flight": "2", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_mon_shutdown_timeout": "5.000000", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_num_cache_shards": "32", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_num_op_tracker_shard": "32", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_numa_auto_affinity": "true", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_numa_node": "-1", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_numa_prefer_iface": "true", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_object_clean_region_max_num_intervals": "10", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_objecter_finishers": "1", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_objectstore": "bluestore", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_objectstore_fuse": "false", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_objectstore_ideal_list_max": "64", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_objectstore_tracing": "false", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_complaint_time": "30.000000", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_history_duration": "600", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_history_size": "20", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_history_slow_op_size": "20", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_log_threshold": "5", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_num_shards": "0", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_num_shards_hdd": "1", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_num_shards_ssd": "8", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_num_threads_per_shard": "0", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_num_threads_per_shard_hdd": "5", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_pq_min_cost": "65536", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_queue": "mclock_scheduler", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_queue_cut_off": "high", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_thread_suicide_timeout": "150", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_op_thread_timeout": "15", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_open_classes_on_start": "true", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_os_flags": "0", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_peering_op_priority": "255", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pg_delete_cost": "1048576", 2026-02-25T08:41:29.252 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pg_delete_priority": "5", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pg_log_dups_tracked": "3000", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pg_log_trim_max": "10000", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pg_log_trim_min": "100", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pg_object_context_cache_count": "64", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_cache_min_evict_age": "0", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_cache_min_flush_age": "0", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_crimson": "true", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_crush_rule": "-1", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_ec_fast_read": "false", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_erasure_code_profile": "plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_flag_bulk": "false", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_flag_ec_optimizations": "false", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_flag_hashpspool": "true", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_flag_nodelete": "false", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_flag_nopgchange": "false", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_flag_nosizechange": "false", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_flags": "0", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_min_size": "2", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_pg_num": "32", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_pgp_num": "0", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_read_ratio": "70", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_size": "3", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_default_type": "replicated", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_erasure_code_stripe_unit": "0", 2026-02-25T08:41:29.253 INFO:teuthology.orchestra.run.trial169.stdout: "osd_pool_use_gmt_hitset": "true", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_push_per_object_cost": "1000", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_read_ec_check_for_errors": "false", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recover_clone_overlap": "true", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recover_clone_overlap_limit": "10", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_cost": "20971520", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_delay_start": "0.000000", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_max_active": "0", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_max_active_hdd": "3", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_max_active_ssd": "10", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_max_chunk": "8388608", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_max_single_start": "1", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_op_priority": "3", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_op_warn_multiple": "16", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_priority": "5", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_retry_interval": "30.000000", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_sleep": "0.000000", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_sleep_degraded": "0.000000", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_sleep_degraded_hdd": "0.100000", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_sleep_degraded_hybrid": "0.025000", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_sleep_degraded_ssd": "0.000000", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_sleep_hdd": "0.100000", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_recovery_sleep_ssd": "0.000000", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_requested_scrub_priority": "5", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_rollback_to_cluster_snap": "", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_auto_repair": "false", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_auto_repair_num_errors": "5", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_backoff_ratio": "0.660000", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_begin_hour": "0", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_begin_week_day": "0", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_chunk_max": "15", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_chunk_min": "5", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_cost": "52428800", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_disable_reservation_queuing": "false", 2026-02-25T08:41:29.254 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_during_recovery": "false", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_end_hour": "0", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_end_week_day": "0", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_event_cost": "4096", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_extended_sleep": "0.000000", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_invalid_stats": "true", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_load_threshold": "10.000000", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_max_interval": "604800.000000", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_max_preemptions": "5", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_min_interval": "86400.000000", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_priority": "5", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_retry_after_noscrub": "60", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_retry_delay": "30", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_retry_new_interval": "10", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_retry_pg_state": "60", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_retry_trimming": "10", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_scrub_sleep": "0.000000", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_shallow_scrub_chunk_max": "100", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_shallow_scrub_chunk_min": "50", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_shutdown_pgref_assert": "false", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_skip_check_past_interval_bounds": "false", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_skip_data_digest": "false", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_smart_report_timeout": "5", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_snap_trim_cost": "1048576", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_snap_trim_priority": "5", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_snap_trim_sleep": "0.000000", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_stats_update_period_not_scrubbing": "120", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_stats_update_period_scrubbing": "15", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_target_transaction_size": "30", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_tier_default_cache_hit_set_count": "4", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2026-02-25T08:41:29.255 INFO:teuthology.orchestra.run.trial169.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "osd_tier_default_cache_mode": "writeback", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "osd_tier_promote_max_objects_sec": "25", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "osd_tracing": "false", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "osd_use_stale_snap": "false", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "osdc_blkin_trace_all": "false", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "paxos_kill_at": "0", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "paxos_max_join_drift": "10", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "paxos_min": "500", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "paxos_min_wait": "0.050000", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "paxos_propose_interval": "1.000000", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "paxos_service_trim_max": "500", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "paxos_service_trim_max_multiplier": "20", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "paxos_service_trim_min": "10", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "paxos_stash_full_interval": "25", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "paxos_trim_max": "500", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "paxos_trim_min": "250", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "perf": "true", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "plugin_crypto_accelerator": "crypto_isal", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "plugin_dir": "/usr/lib64/ceph", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "pool_availability_update_interval": "1.000000", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "public_addr": "-", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "public_addrv": "", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "public_bind_addr": "-", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "public_network": "", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "public_network_interface": "", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "qat_compressor_busy_polling": "false", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "qat_compressor_enabled": "false", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "qat_compressor_session_max_number": "256", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "rados_mon_op_timeout": "0", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "rados_osd_op_timeout": "0", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "rados_replica_read_policy": "default", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "rados_replica_read_policy_on_objclass": "false", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "rados_tracing": "false", 2026-02-25T08:41:29.256 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_atime_update_interval": "60", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_balance_parent_reads": "false", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_balance_snap_reads": "false", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_blkin_trace_all": "false", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_blocklist_expire_seconds": "0", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_blocklist_on_break_lock": "true", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_cache": "true", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_cache_block_writes_upfront": "false", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_cache_max_dirty": "25165824", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_cache_max_dirty_age": "1.000000", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_cache_max_dirty_object": "0", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_cache_policy": "writearound", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_cache_size": "33554432", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_cache_target_dirty": "16777216", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_cache_writethrough_until_flush": "true", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_clone_copy_on_read": "false", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_compression_hint": "none", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_concurrent_management_ops": "10", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_config_pool_override_update_timestamp": "0", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_default_clone_format": "auto", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_default_data_pool": "", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_default_features": "61", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_default_format": "2", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_default_map_options": "", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_default_order": "22", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_default_pool": "rbd", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_default_stripe_count": "0", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_default_stripe_unit": "0", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_disable_zero_copy_writes": "true", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_discard_granularity_bytes": "65536", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_discard_on_zeroed_write_same": "true", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_enable_alloc_hint": "true", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_io_scheduler": "simple", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_journal_commit_age": "5.000000", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2026-02-25T08:41:29.257 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_journal_max_payload_bytes": "16384", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_journal_object_flush_age": "0.000000", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_journal_object_flush_bytes": "1048576", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_journal_object_flush_interval": "0", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_journal_order": "24", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_journal_pool": "", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_journal_splay_width": "4", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_localize_parent_reads": "false", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_localize_snap_reads": "false", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_image_policy_type": "simple", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_image_state_check_interval": "30", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_memory_autotune": "true", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_memory_base": "805306368", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_memory_cache_min": "134217728", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_memory_target": "4294967296", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_perf_stats_prio": "5", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirroring_delete_delay": "0", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirroring_replay_delay": "0", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2026-02-25T08:41:29.258 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_move_to_trash_on_remove": "false", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_mtime_update_interval": "60", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_non_blocking_aio": "true", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_op_thread_timeout": "60", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_op_threads": "1", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_parent_cache_enabled": "false", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_persistent_cache_mode": "disabled", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_persistent_cache_path": "/tmp", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_persistent_cache_size": "1073741824", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_plugins": "", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_bps_burst": "0", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_bps_burst_seconds": "1", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_bps_limit": "0", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_exclude_ops": "0", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_iops_burst": "0", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_iops_burst_seconds": "1", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_iops_limit": "0", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_read_bps_burst": "0", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_read_bps_limit": "0", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_read_iops_burst": "0", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_read_iops_limit": "0", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_schedule_tick_min": "50", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_write_bps_burst": "0", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_write_bps_limit": "0", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_write_iops_burst": "0", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_qos_write_iops_limit": "0", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_quiesce_notification_attempts": "10", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_read_from_replica_policy": "default", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_readahead_max_bytes": "524288", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_readahead_trigger_requests": "10", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_request_timed_out_seconds": "30", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_skip_partial_discard": "true", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2026-02-25T08:41:29.259 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_tracing": "false", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_validate_names": "true", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rbd_validate_pool": "true", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "restapi_base_url": "", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "restapi_log_level": "", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_account_default_quota_max_objects": "-1", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_account_default_quota_max_size": "-1", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_acl_grants_max_num": "100", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_admin_entry": "admin", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_asio_assert_yielding": "false", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_backend_store": "rados", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_barbican_url": "", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_beast_enable_async": "true", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_bucket_counters_cache": "false", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_bucket_counters_cache_size": "10000", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_bucket_default_quota_max_size": "-1", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_bucket_eexist_override": "false", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_bucket_index_max_aio": "128", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_bucket_logging_obj_roll_time": "300", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_bucket_persistent_notif_num_shards": "11", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_bucket_quota_cache_size": "10000", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_bucket_quota_ttl": "600", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_bucket_sync_spawn_window": "20", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_cache_enabled": "true", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_cache_expiry_interval": "900", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_cache_lru_size": "25000", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_config_store": "rados", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_content_length_compat": "false", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_copy_obj_progress": "true", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_cors_rules_max_num": "100", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_cross_domain_policy": "", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_default_encryption_key": "", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_kmip_addr": "", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_kmip_ca_path": "", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_kmip_client_cert": "", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_kmip_client_key": "", 2026-02-25T08:41:29.260 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_kmip_kms_key_template": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_kmip_password": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_kmip_username": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_require_ssl": "true", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_sse_s3_backend": "vault", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_suppress_logs": "true", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_vault_addr": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_vault_auth": "token", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_vault_namespace": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_vault_prefix": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_vault_secret_engine": "transit", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_vault_ssl_cacert": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_vault_token_file": "", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_crypt_vault_verify_ssl": "true", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_curl_buffersize": "524288", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_curl_low_speed_limit": "1024", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_curl_low_speed_time": "300", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_curl_tcp_keepalive": "0", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_curl_wait_timeout_ms": "1000", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d3n_l1_fadvise": "4", 2026-02-25T08:41:29.261 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d3n_libaio_aio_num": "64", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d3n_libaio_aio_threads": "20", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d4n_backend_address": "127.0.0.1:6379", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d4n_cache_cleaning_interval": "1000", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d4n_l1_datacache_disk_reserve": "1073741824", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d4n_l1_fadvise": "4", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d4n_l1_write_open_flags": "4096", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d4n_libaio_aio_num": "64", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d4n_libaio_aio_threads": "20", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_d4n_local_rgw_address": "127.0.0.1:8000", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_data_log_changes_size": "1000", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_data_log_num_shards": "128", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_data_log_window": "30", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_data_notify_interval_msec": "0", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_data_sync_poll_interval": "20", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_data_sync_spawn_window": "20", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_debug_inject_set_olh_err": "0", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_default_data_log_backing": "fifo", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_default_realm_info_oid": "default.realm", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_default_region_info_oid": "default.region", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_default_zone_info_oid": "default.zone", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_defer_to_bucket_acls": "", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_delete_multi_obj_max_num": "1000", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_disable_s3select": "false", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dmclock_admin_lim": "0.000000", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dmclock_admin_res": "100.000000", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dmclock_auth_lim": "0.000000", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dmclock_auth_res": "200.000000", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dmclock_data_lim": "0.000000", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dmclock_data_res": "500.000000", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dmclock_data_wgt": "500.000000", 2026-02-25T08:41:29.262 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dmclock_metadata_res": "500.000000", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dns_name": "", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dns_s3website_name": "", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dynamic_resharding": "true", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_enable_gc_threads": "true", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_enable_jwks_url_verification": "false", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_enable_lc_threads": "true", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_enable_mdsearch": "true", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_enable_ops_log": "false", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_enable_quota_threads": "true", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_enable_restore_threads": "true", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_enable_static_website": "false", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_enable_usage_log": "false", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_enforce_swift_acls": "true", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_exit_timeout_secs": "120", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_expose_bucket": "false", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_extended_http_attrs": "", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_filter": "none", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_frontends": "beast port=7480", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_gc_max_concurrent_io": "10", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_gc_max_deferred": "50", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_gc_max_objs": "32", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_gc_max_queue_size": "134213632", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_gc_max_trim_chunk": "16", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_gc_obj_min_wait": "7200", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_gc_processor_max_time": "3600", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_gc_processor_period": "3600", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_get_obj_max_req_size": "4194304", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_get_obj_window_size": "16777216", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_graceful_stop": "false", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_healthcheck_disabling_path": "", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_http_notif_connection_timeout": "5", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_http_notif_max_inflight": "8192", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_http_notif_message_timeout": "10", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_ignore_get_invalid_range": "false", 2026-02-25T08:41:29.263 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_init_timeout": "300", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_kafka_connection_idle": "300", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_kafka_message_timeout": "5000", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_kafka_sleep_timeout": "10", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_accepted_admin_roles": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_accepted_reader_roles": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_admin_domain": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_admin_password": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_admin_password_path": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_admin_project": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_admin_tenant": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_admin_user": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_barbican_domain": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_barbican_password": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_barbican_project": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_barbican_tenant": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_barbican_user": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_implicit_tenants": "false", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_service_token_enabled": "false", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_token_cache_size": "10000", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_token_cache_ttl": "300", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_url": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_keystone_verify_ssl": "true", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_lc_debug_interval": "-1", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_lc_list_cnt": "1000", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_lc_lock_max_time": "90", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_lc_max_objs": "32", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_lc_max_rules": "1000", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_lc_max_worker": "3", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_lc_max_wp_worker": "128", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_lc_ordered_list_threshold": "500", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_lc_thread_delay": "0", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_ldap_dnattr": "uid", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_ldap_searchfilter": "", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2026-02-25T08:41:29.264 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_ldap_uri": "", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_lfuda_sync_frequency": "60", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_list_bucket_min_readahead": "1000", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_list_buckets_max_chunk": "1000", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_log_http_headers": "", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_log_nonexistent_bucket": "false", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_log_object_name_utc": "false", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_lua_enable": "true", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_lua_max_memory_per_state": "128000", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_lua_max_runtime_per_state": "1000", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_max_attr_name_len": "0", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_max_attr_size": "0", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_max_attrs_num_in_req": "0", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_max_chunk_size": "4194304", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_max_concurrent_requests": "1024", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_max_control_aio": "8", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_max_dynamic_shards": "1999", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_max_listing_results": "5000", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_max_notify_retries": "10", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_max_objs_per_shard": "100000", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_max_put_param_size": "1048576", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_max_put_size": "5368709120", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_max_slo_entries": "1000", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_md_log_max_shards": "64", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_md_notify_interval_msec": "200", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_meta_sync_poll_interval": "20", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_meta_sync_spawn_window": "20", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_mime_types_file": "/etc/mime.types", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_mp_lock_max_time": "600", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_multi_obj_del_max_aio": "16", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_multipart_min_part_size": "5242880", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_multipart_part_upload_limit": "10000", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_nfs_fhcache_partitions": "3", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_nfs_fhcache_size": "2017", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_nfs_frontends": "rgw-nfs", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_nfs_lru_lanes": "5", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_nfs_max_gc": "300", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_nfs_namespace_expire_secs": "300", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_nfs_run_gc_threads": "false", 2026-02-25T08:41:29.265 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_nfs_run_lc_threads": "false", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_nfs_run_quota_threads": "false", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_nfs_run_restore_threads": "false", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_nfs_run_sync_thread": "false", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_nfs_s3_fast_attrs": "false", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_nfs_write_completion_interval_s": "10", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_num_async_rados_threads": "32", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_num_control_oids": "8", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_numa_node": "-1", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_obj_stripe_size": "4194304", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_obj_tombstone_cache_size": "1000", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_objexp_chunk_size": "100", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_objexp_gc_interval": "600", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_objexp_hints_num_shards": "127", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_olh_pending_timeout_sec": "3600", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_op_thread_suicide_timeout": "0", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_op_thread_timeout": "600", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_op_tracing": "false", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_opa_token": "", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_opa_url": "", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_opa_verify_ssl": "true", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_ops_log_data_backlog": "5242880", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_ops_log_rados": "false", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_ops_log_socket_path": "", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_override_bucket_index_max_shards": "0", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_parquet_buffer_size": "16777216", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_period_push_interval": "2.000000", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_period_push_interval_max": "30.000000", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_period_root_pool": ".rgw.root", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_policy_reject_invalid_principals": "true", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_posix_cache_lanes": "3", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_posix_cache_lmdb_count": "3", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_posix_cache_max_buckets": "100", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_posix_cache_partitions": "3", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_posix_userdb_dir": "/var/lib/ceph/radosgw", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_print_continue": "true", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_print_prohibited_content_length": "false", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_put_obj_max_window_size": "67108864", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_put_obj_min_window_size": "16777216", 2026-02-25T08:41:29.266 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_rados_pool_recovery_priority": "5", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_rados_tracing": "false", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_ratelimit_interval": "60", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_realm": "", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_realm_id": "", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_realm_root_pool": ".rgw.root", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_redis_connection_pool_size": "512", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_region": "", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_region_root_pool": ".rgw.root", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_relaxed_region_enforcement": "false", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_relaxed_s3_bucket_names": "false", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_request_uri": "", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_reshard_batch_size": "64", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_reshard_bucket_lock_duration": "360", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_reshard_debug_interval": "-1", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_reshard_max_aio": "128", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_reshard_num_logs": "16", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_reshard_progress_judge_interval": "120", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_reshard_progress_judge_ratio": "0.500000", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_reshard_thread_interval": "600", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_reshardlog_threshold": "30000", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_resolve_cname": "false", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_rest_getusage_op_compat": "false", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_restore_debug_interval": "-1", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_restore_lock_max_time": "90", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_restore_max_objs": "32", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_restore_processor_period": "900", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_run_sync_thread": "true", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_s3_auth_disable_signature_url": "false", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_s3_auth_order": "sts, external, local", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_s3_auth_use_keystone": "false", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_s3_auth_use_ldap": "false", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_s3_auth_use_rados": "true", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_s3_auth_use_sts": "false", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_s3_client_max_sig_ver": "-1", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_s3_success_create_obj_status": "0", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_safe_max_objects_per_shard": "102400", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_scheduler_type": "throttler", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_script_uri": "", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_service_provider_name": "", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_shard_warning_threshold": "90.000000", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sts_client_id": "", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sts_client_secret": "", 2026-02-25T08:41:29.267 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sts_entry": "sts", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sts_key": "", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sts_max_session_duration": "43200", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sts_min_session_duration": "900", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sts_token_introspection_url": "", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_swift_account_in_url": "false", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_swift_auth_entry": "auth", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_swift_auth_url": "", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_swift_custom_header": "", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_swift_enforce_content_length": "false", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_swift_need_stats": "true", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_swift_tenant_name": "", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_swift_token_expiration": "86400", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_swift_url": "", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_swift_url_prefix": "swift", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_swift_versioning_enabled": "false", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sync_lease_period": "120", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sync_log_trim_interval": "1200", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sync_log_trim_max_buckets": "16", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sync_obj_etag_verify": "false", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sync_trace_history_size": "4096", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sync_trace_per_node_log_size": "32", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_thread_pool_size": "512", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_topic_persistency_max_retries": "0", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_topic_persistency_sleep_duration": "0", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_topic_persistency_time_to_live": "0", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_topic_require_publish_policy": "false", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_torrent_comment": "", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_torrent_createby": "", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_torrent_encoding": "", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_torrent_flag": "false", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_torrent_max_size": "5368709120", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_torrent_origin": "", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_torrent_sha_unit": "524288", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_torrent_tracker": "", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_trust_forwarded_https": "false", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_usage_log_flush_threshold": "1024", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_usage_log_key_transition": "true", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_usage_log_tick_interval": "30", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_usage_max_shards": "32", 2026-02-25T08:41:29.268 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_usage_max_user_shards": "1", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_use_opa_authz": "false", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_user_counters_cache": "false", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_user_counters_cache_size": "10000", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_user_default_quota_max_objects": "-1", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_user_default_quota_max_size": "-1", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_user_max_buckets": "1000", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_user_policies_max_num": "100", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_user_quota_sync_idle_users": "false", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_user_quota_sync_interval": "86400", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_user_quota_sync_wait_time": "86400", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_user_unique_email": "true", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_verify_ssl": "true", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_website_routing_rules_max_num": "50", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_zone": "", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_zone_id": "", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_zone_root_pool": ".rgw.root", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_zonegroup": "", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_zonegroup_id": "", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgwlc_auto_session_clear": "true", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rgwlc_skip_bucket_step": "false", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_block_size": "4096", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_bloom_bits_per_key": "20", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_cache_row_ratio": "0.000000", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_cache_shard_bits": "4", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_cache_size": "536870912", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_cache_type": "binned_lru", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_cf_compact_on_deletion": "true", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_collect_compaction_stats": "false", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_collect_extended_stats": "false", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_collect_memory_stats": "false", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_delete_range_threshold": "1048576", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_index_type": "binary_search", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_log_to_ceph_log": "true", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_metadata_block_size": "4096", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_partition_filters": "false", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_perf": "false", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rotating_keys_bootstrap_timeout": "30", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "rotating_keys_renewal_timeout": "10", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "run_dir": "/var/run/ceph", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_block_create": "true", 2026-02-25T08:41:29.269 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_cachepin_2q_in_ratio": "0.500000", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_cachepin_2q_out_ratio": "0.500000", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_cachepin_size_pershard": "2147483648", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_cachepin_type": "LRU", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_cbjournal_size": "5368709120", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_cold_tier_generations": "3", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_data_delta_based_overwrite": "0", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_default_max_object_size": "16777216", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_default_object_metadata_reservation": "16777216", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_device_size": "53687091200", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_disable_end_to_end_data_protection": "true", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_full_integrity_check": "false", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_hot_tier_generations": "5", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_journal_batch_capacity": "16", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_journal_batch_flush_size": "16777216", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_journal_iodepth_limit": "5", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_main_device_type": "SSD", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_max_concurrent_transactions": "128", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_max_data_allocation_size": "0", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_segment_cleaner_gc_formula": "cost_benefit", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "seastore_segment_size": "67108864", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "service_unique_id": "", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "set_keepcaps": "false", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "setgroup": "", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "setuser": "", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "setuser_match_path": "", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "subv_metrics_window_interval": "30", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "target_max_misplaced_ratio": "0.050000", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "thp": "false", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "threadpool_default_timeout": "60", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "threadpool_empty_queue_max_wait": "2", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "throttler_perf_counter": "true", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "tmp_dir": "/tmp", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "uadk_compressor_enabled": "false", 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout: "uadk_wd_sync_ctx_num": "2" 2026-02-25T08:41:29.270 INFO:teuthology.orchestra.run.trial169.stdout:} 2026-02-25T08:41:29.272 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2026-02-25T08:41:29.346 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:29.344+0000 7fef2c10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:29.349 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:29.348+0000 7fef2c10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:29.358 INFO:teuthology.orchestra.run.trial169.stdout:{} 2026-02-25T08:41:29.358 INFO:teuthology.orchestra.run.trial169.stderr:mon_osd_down_out_interval = '' (not observed, change may require restart) 2026-02-25T08:41:29.365 DEBUG:teuthology.orchestra.run.trial169:> type ceph-objectstore-tool 2026-02-25T08:41:29.380 INFO:teuthology.orchestra.run.trial169.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2026-02-25T08:41:29.381 DEBUG:teuthology.orchestra.run.trial183:> type ceph-objectstore-tool 2026-02-25T08:41:29.397 INFO:teuthology.orchestra.run.trial183.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2026-02-25T08:41:29.398 INFO:teuthology.run_tasks:Running task rados... 2026-02-25T08:41:29.420 INFO:tasks.rados:Beginning rados... 2026-02-25T08:41:29.420 INFO:tasks.rados:config is {'balance_reads': True, 'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2026-02-25T08:41:29.420 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-logstoremanager'}, 'ceph': {'conf': {'global': {'enable experimental unrecoverable data corrupting features': 'crimson', 'ms client mode': 'crc', 'ms cluster mode': 'crc', 'ms mon client mode': 'crc', 'ms mon cluster mode': 'crc', 'ms mon service mode': 'crc', 'ms service mode': 'crc', 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'osd pool default crimson': True, 'paxos service trim min': 10}, 'osd': {'crimson cpu num': 4, 'crimson osd obc lru size': 10, 'debug ms': 20, 'debug osd': 20, 'debug seastore': 20, 'debug seastore backref': 20, 'debug seastore cache': 20, 'debug seastore cleaner': 20, 'debug seastore device': 20, 'debug seastore epm': 20, 'debug seastore fixedkv tree': 20, 'debug seastore journal': 20, 'debug seastore lba': 20, 'debug seastore odata': 20, 'debug seastore omap': 20, 'debug seastore onode': 20, 'debug seastore t': 20, 'debug seastore tm': 20, 'osd blocked scrub grace period': 3600, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd max markdown count': 1000, 'osd objectstore': 'seastore', 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2, 'osd target transaction size': 5, 'seastore cachepin size pershard': '64M', 'seastore main device type': 'RANDOM_BLOCK_SSD', 'seastore max concurrent transactions': 8}}, 'crimson_compat': True, 'flavor': 'debug', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', 'SLOW_OPS', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub', 'Degraded data redundancy', 'is down', 'osds down', 'pg .*? is .*?degraded.*?, acting', 'pg .*? is stuck', 'pg degraded', 'PG_BACKFILL_FULL', 'Low space hindering backfill .*? backfill_toofull', 'OSD_ROOT_DOWN', 'pgs degraded', 'pgs undersized', 'is active.*backfill_toofull.*'], 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'debug', 'sha1': '6aeb6975bc9a4790a399c9cc70dba9635d2a7819'}}, 'thrashosds': {'min_in': 4}, 'workunit': {'branch': 'main', 'env': {'CRIMSON_COMPAT': '1'}, 'sha1': 'c32469671c6b0222f786dd36c34eb702f780a11d'}} 2026-02-25T08:41:29.420 INFO:tasks.rados:config is {'balance_reads': True, 'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 0, 'delete': 50, 'read': 100, 'rollback': 0, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2026-02-25T08:41:29.421 DEBUG:teuthology.run_tasks:Unwinding manager rados 2026-02-25T08:41:29.435 INFO:tasks.rados:joining rados 2026-02-25T08:41:29.435 INFO:tasks.thrashosds.thrasher:starting do_thrash 2026-02-25T08:41:29.435 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-25T08:41:29.436 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2026-02-25T08:41:29.436 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2026-02-25T08:41:29.437 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:41:29.437 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2026-02-25T08:41:29.437 INFO:tasks.rados:starting run 0 out of 1 2026-02-25T08:41:29.437 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2026-02-25T08:41:29.437 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2026-02-25T08:41:29.517 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:29.514+0000 7f95ddd64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:29.518 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:29.514+0000 7f4d671ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:29.519 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:29.518+0000 7f95ddd64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:29.520 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:29.514+0000 7f39228cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:29.521 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:29.520+0000 7f4d671ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:29.522 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:29.521+0000 7f39228cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:30.488 INFO:teuthology.orchestra.run.trial169.stderr:pool 'unique_pool_0' created 2026-02-25T08:41:30.488 INFO:teuthology.orchestra.run.trial169.stderr:reweighted osd.0 to 1 (10000) 2026-02-25T08:41:30.502 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2026-02-25T08:41:30.503 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2026-02-25T08:41:30.623 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:30.618+0000 7f86f561a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:30.624 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:30.617+0000 7fc55ba46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:30.626 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:30.624+0000 7f86f561a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:30.627 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:30.626+0000 7fc55a7bd640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.492 INFO:teuthology.orchestra.run.trial169.stderr:enabled application 'rados' on pool 'unique_pool_0' 2026-02-25T08:41:31.493 INFO:teuthology.orchestra.run.trial169.stderr:reweighted osd.1 to 1 (10000) 2026-02-25T08:41:31.506 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2026-02-25T08:41:31.583 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:41:31.631 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:31.626+0000 7f2307ba4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.634 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:31.633+0000 7f2307ba4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.667 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:31.665+0000 7f5b66ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:31.670 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:31.668+0000 7f5b66ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:32.496 INFO:teuthology.orchestra.run.trial169.stderr:reweighted osd.2 to 1 (10000) 2026-02-25T08:41:32.508 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pct_update_delay 5 2026-02-25T08:41:32.512 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2026-02-25T08:41:32.589 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:32.584+0000 7fc7916cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:32.592 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:32.591+0000 7fc7916cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:32.630 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:32.626+0000 7f2b25235640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:32.633 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:32.632+0000 7f2b25235640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:33.500 INFO:teuthology.orchestra.run.trial169.stderr:set pool 3 pct_update_delay to 5 2026-02-25T08:41:33.501 INFO:teuthology.orchestra.run.trial169.stderr:reweighted osd.3 to 1 (10000) 2026-02-25T08:41:33.515 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2026-02-25T08:41:33.515 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:33.517 DEBUG:teuthology.orchestra.run.trial169:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph_test_rados --balance-reads --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 0 --op copy_from 0 --op write_excl 50 --pool unique_pool_0 2026-02-25T08:41:33.603 INFO:tasks.rados.rados.0.trial169.stdout:adding op weight read -> 100 2026-02-25T08:41:33.604 INFO:tasks.rados.rados.0.trial169.stdout:adding op weight write -> 50 2026-02-25T08:41:33.604 INFO:tasks.rados.rados.0.trial169.stdout:adding op weight delete -> 50 2026-02-25T08:41:33.604 INFO:tasks.rados.rados.0.trial169.stdout:adding op weight snap_create -> 50 2026-02-25T08:41:33.604 INFO:tasks.rados.rados.0.trial169.stdout:adding op weight snap_remove -> 50 2026-02-25T08:41:33.604 INFO:tasks.rados.rados.0.trial169.stdout:adding op weight write_excl -> 50 2026-02-25T08:41:33.604 INFO:tasks.rados.rados.0.trial169.stdout:ceph version 20.3.0-5338-g6aeb6975 (6aeb6975bc9a4790a399c9cc70dba9635d2a7819) tentacle (dev - Debug) 2026-02-25T08:41:33.604 INFO:tasks.rados.rados.0.trial169.stdout:Configuration: 2026-02-25T08:41:33.604 INFO:tasks.rados.rados.0.trial169.stdout: Number of operations: 4000 2026-02-25T08:41:33.604 INFO:tasks.rados.rados.0.trial169.stdout: Number of objects: 50 2026-02-25T08:41:33.605 INFO:tasks.rados.rados.0.trial169.stdout: Max in flight operations: 16 2026-02-25T08:41:33.605 INFO:tasks.rados.rados.0.trial169.stdout: Object size (in bytes): 4000000 2026-02-25T08:41:33.605 INFO:tasks.rados.rados.0.trial169.stdout: Write stride min: 400000 2026-02-25T08:41:33.605 INFO:tasks.rados.rados.0.trial169.stdout: Write stride max: 800000 2026-02-25T08:41:33.634 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:33.630+0000 7f861275b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:33.636 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:33.634+0000 7f861275b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:33.732 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:41:33.749 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:33.750 INFO:teuthology.orchestra.run.trial169.stderr:dumped all 2026-02-25T08:41:33.758 INFO:teuthology.orchestra.run.trial169.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2026-02-25T08:41:33.499834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83891320,"kb_used_data":83891320,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125823816,"statfs":{"total":214748299264,"available":128843587584,"internally_reserved":0,"allocated":85904711680,"data_stored":85904711680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.173910"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.580852+0000","last_change":"2026-02-25T08:41:31.582015+0000","last_active":"2026-02-25T08:41:31.580852+0000","last_peered":"2026-02-25T08:41:31.580852+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.577339+0000","last_became_peered":"2026-02-25T08:41:31.577339+0000","last_unstale":"2026-02-25T08:41:31.580852+0000","last_undegraded":"2026-02-25T08:41:31.580852+0000","last_fullsized":"2026-02-25T08:41:31.580852+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.582809+0000","last_change":"2026-02-25T08:41:31.583321+0000","last_active":"2026-02-25T08:41:31.582809+0000","last_peered":"2026-02-25T08:41:31.582809+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.567579+0000","last_became_peered":"2026-02-25T08:41:31.567579+0000","last_unstale":"2026-02-25T08:41:31.582809+0000","last_undegraded":"2026-02-25T08:41:31.582809+0000","last_fullsized":"2026-02-25T08:41:31.582809+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.568542+0000","last_change":"2026-02-25T08:41:31.569490+0000","last_active":"2026-02-25T08:41:31.568542+0000","last_peered":"2026-02-25T08:41:31.568542+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.550562+0000","last_became_peered":"2026-02-25T08:41:31.550562+0000","last_unstale":"2026-02-25T08:41:31.568542+0000","last_undegraded":"2026-02-25T08:41:31.568542+0000","last_fullsized":"2026-02-25T08:41:31.568542+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.583197+0000","last_change":"2026-02-25T08:41:31.583740+0000","last_active":"2026-02-25T08:41:31.583197+0000","last_peered":"2026-02-25T08:41:31.583197+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.571602+0000","last_became_peered":"2026-02-25T08:41:31.571602+0000","last_unstale":"2026-02-25T08:41:31.583197+0000","last_undegraded":"2026-02-25T08:41:31.583197+0000","last_fullsized":"2026-02-25T08:41:31.583197+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1,"reported_epoch":16,"state":"creating+peering","last_fresh":"2026-02-25T08:41:30.522282+0000","last_change":"2026-02-25T08:41:30.522282+0000","last_active":"2026-02-25T08:41:30.484777+0000","last_peered":"2026-02-25T08:41:30.484777+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-25T08:41:30.522282+0000","last_undegraded":"2026-02-25T08:41:30.522282+0000","last_fullsized":"2026-02-25T08:41:30.522282+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.587533+0000","last_change":"2026-02-25T08:41:31.587841+0000","last_active":"2026-02-25T08:41:31.587533+0000","last_peered":"2026-02-25T08:41:31.587533+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.576273+0000","last_became_peered":"2026-02-25T08:41:31.576273+0000","last_unstale":"2026-02-25T08:41:31.587533+0000","last_undegraded":"2026-02-25T08:41:31.587533+0000","last_fullsized":"2026-02-25T08:41:31.587533+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.580842+0000","last_change":"2026-02-25T08:41:31.581547+0000","last_active":"2026-02-25T08:41:31.580842+0000","last_peered":"2026-02-25T08:41:31.580842+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.569266+0000","last_became_peered":"2026-02-25T08:41:31.569266+0000","last_unstale":"2026-02-25T08:41:31.580842+0000","last_undegraded":"2026-02-25T08:41:31.580842+0000","last_fullsized":"2026-02-25T08:41:31.580842+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1,"reported_epoch":16,"state":"creating+peering","last_fresh":"2026-02-25T08:41:30.511878+0000","last_change":"2026-02-25T08:41:30.511878+0000","last_active":"2026-02-25T08:41:30.484777+0000","last_peered":"2026-02-25T08:41:30.484777+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-25T08:41:30.511878+0000","last_undegraded":"2026-02-25T08:41:30.511878+0000","last_fullsized":"2026-02-25T08:41:30.511878+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.507232+0000","last_change":"2026-02-25T08:41:26.373872+0000","last_active":"2026-02-25T08:41:31.507232+0000","last_peered":"2026-02-25T08:41:31.507232+0000","last_clean":"2026-02-25T08:41:31.507232+0000","last_became_active":"2026-02-25T08:41:24.399215+0000","last_became_peered":"2026-02-25T08:41:24.399215+0000","last_unstale":"2026-02-25T08:41:31.507232+0000","last_undegraded":"2026-02-25T08:41:31.507232+0000","last_fullsized":"2026-02-25T08:41:31.507232+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1,"reported_epoch":16,"state":"creating+peering","last_fresh":"2026-02-25T08:41:30.501449+0000","last_change":"2026-02-25T08:41:30.501449+0000","last_active":"2026-02-25T08:41:30.484777+0000","last_peered":"2026-02-25T08:41:30.484777+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-25T08:41:30.501449+0000","last_undegraded":"2026-02-25T08:41:30.501449+0000","last_fullsized":"2026-02-25T08:41:30.501449+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.512406+0000","last_change":"2026-02-25T08:41:26.374903+0000","last_active":"2026-02-25T08:41:31.512406+0000","last_peered":"2026-02-25T08:41:31.512406+0000","last_clean":"2026-02-25T08:41:31.512406+0000","last_became_active":"2026-02-25T08:41:24.402449+0000","last_became_peered":"2026-02-25T08:41:24.402449+0000","last_unstale":"2026-02-25T08:41:31.512406+0000","last_undegraded":"2026-02-25T08:41:31.512406+0000","last_fullsized":"2026-02-25T08:41:31.512406+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.596834+0000","last_change":"2026-02-25T08:41:31.597333+0000","last_active":"2026-02-25T08:41:31.596834+0000","last_peered":"2026-02-25T08:41:31.596834+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.591938+0000","last_became_peered":"2026-02-25T08:41:31.591938+0000","last_unstale":"2026-02-25T08:41:31.596834+0000","last_undegraded":"2026-02-25T08:41:31.596834+0000","last_fullsized":"2026-02-25T08:41:31.596834+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":24,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.506042+0000","last_change":"2026-02-25T08:41:26.374022+0000","last_active":"2026-02-25T08:41:31.506042+0000","last_peered":"2026-02-25T08:41:31.506042+0000","last_clean":"2026-02-25T08:41:31.506042+0000","last_became_active":"2026-02-25T08:41:24.420798+0000","last_became_peered":"2026-02-25T08:41:24.420798+0000","last_unstale":"2026-02-25T08:41:31.506042+0000","last_undegraded":"2026-02-25T08:41:31.506042+0000","last_fullsized":"2026-02-25T08:41:31.506042+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.627251+0000","last_change":"2026-02-25T08:41:31.627868+0000","last_active":"2026-02-25T08:41:31.627251+0000","last_peered":"2026-02-25T08:41:31.627251+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.623233+0000","last_became_peered":"2026-02-25T08:41:31.623233+0000","last_unstale":"2026-02-25T08:41:31.627251+0000","last_undegraded":"2026-02-25T08:41:31.627251+0000","last_fullsized":"2026-02-25T08:41:31.627251+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.497473+0000","last_change":"2026-02-25T08:41:26.371067+0000","last_active":"2026-02-25T08:41:30.497473+0000","last_peered":"2026-02-25T08:41:30.497473+0000","last_clean":"2026-02-25T08:41:30.497473+0000","last_became_active":"2026-02-25T08:41:24.403313+0000","last_became_peered":"2026-02-25T08:41:24.403313+0000","last_unstale":"2026-02-25T08:41:30.497473+0000","last_undegraded":"2026-02-25T08:41:30.497473+0000","last_fullsized":"2026-02-25T08:41:30.497473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":88,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.506042+0000","last_change":"2026-02-25T08:41:24.427616+0000","last_active":"2026-02-25T08:41:31.506042+0000","last_peered":"2026-02-25T08:41:31.506042+0000","last_clean":"2026-02-25T08:41:31.506042+0000","last_became_active":"2026-02-25T08:41:24.406166+0000","last_became_peered":"2026-02-25T08:41:24.406166+0000","last_unstale":"2026-02-25T08:41:31.506042+0000","last_undegraded":"2026-02-25T08:41:31.506042+0000","last_fullsized":"2026-02-25T08:41:31.506042+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.567233+0000","last_change":"2026-02-25T08:41:31.567773+0000","last_active":"2026-02-25T08:41:31.567233+0000","last_peered":"2026-02-25T08:41:31.567233+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.558932+0000","last_became_peered":"2026-02-25T08:41:31.558932+0000","last_unstale":"2026-02-25T08:41:31.567233+0000","last_undegraded":"2026-02-25T08:41:31.567233+0000","last_fullsized":"2026-02-25T08:41:31.567233+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.608543+0000","last_change":"2026-02-25T08:41:31.609093+0000","last_active":"2026-02-25T08:41:31.608543+0000","last_peered":"2026-02-25T08:41:31.608543+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.602591+0000","last_became_peered":"2026-02-25T08:41:31.602591+0000","last_unstale":"2026-02-25T08:41:31.608543+0000","last_undegraded":"2026-02-25T08:41:31.608543+0000","last_fullsized":"2026-02-25T08:41:31.608543+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.498290+0000","last_change":"2026-02-25T08:41:26.371743+0000","last_active":"2026-02-25T08:41:30.498290+0000","last_peered":"2026-02-25T08:41:30.498290+0000","last_clean":"2026-02-25T08:41:30.498290+0000","last_became_active":"2026-02-25T08:41:24.408398+0000","last_became_peered":"2026-02-25T08:41:24.408398+0000","last_unstale":"2026-02-25T08:41:30.498290+0000","last_undegraded":"2026-02-25T08:41:30.498290+0000","last_fullsized":"2026-02-25T08:41:30.498290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.563878+0000","last_change":"2026-02-25T08:41:31.564506+0000","last_active":"2026-02-25T08:41:31.563878+0000","last_peered":"2026-02-25T08:41:31.563878+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.559738+0000","last_became_peered":"2026-02-25T08:41:31.559738+0000","last_unstale":"2026-02-25T08:41:31.563878+0000","last_undegraded":"2026-02-25T08:41:31.563878+0000","last_fullsized":"2026-02-25T08:41:31.563878+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.508306+0000","last_change":"2026-02-25T08:41:26.375854+0000","last_active":"2026-02-25T08:41:31.508306+0000","last_peered":"2026-02-25T08:41:31.508306+0000","last_clean":"2026-02-25T08:41:31.508306+0000","last_became_active":"2026-02-25T08:41:24.420122+0000","last_became_peered":"2026-02-25T08:41:24.420122+0000","last_unstale":"2026-02-25T08:41:31.508306+0000","last_undegraded":"2026-02-25T08:41:31.508306+0000","last_fullsized":"2026-02-25T08:41:31.508306+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.557289+0000","last_change":"2026-02-25T08:41:31.558232+0000","last_active":"2026-02-25T08:41:31.557289+0000","last_peered":"2026-02-25T08:41:31.557289+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.548939+0000","last_became_peered":"2026-02-25T08:41:31.548939+0000","last_unstale":"2026-02-25T08:41:31.557289+0000","last_undegraded":"2026-02-25T08:41:31.557289+0000","last_fullsized":"2026-02-25T08:41:31.557289+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.507371+0000","last_change":"2026-02-25T08:41:26.377253+0000","last_active":"2026-02-25T08:41:31.507371+0000","last_peered":"2026-02-25T08:41:31.507371+0000","last_clean":"2026-02-25T08:41:31.507371+0000","last_became_active":"2026-02-25T08:41:24.409565+0000","last_became_peered":"2026-02-25T08:41:24.409565+0000","last_unstale":"2026-02-25T08:41:31.507371+0000","last_undegraded":"2026-02-25T08:41:31.507371+0000","last_fullsized":"2026-02-25T08:41:31.507371+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.592738+0000","last_change":"2026-02-25T08:41:31.593574+0000","last_active":"2026-02-25T08:41:31.592738+0000","last_peered":"2026-02-25T08:41:31.592738+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.586165+0000","last_became_peered":"2026-02-25T08:41:31.586165+0000","last_unstale":"2026-02-25T08:41:31.592738+0000","last_undegraded":"2026-02-25T08:41:31.592738+0000","last_fullsized":"2026-02-25T08:41:31.592738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.507032+0000","last_change":"2026-02-25T08:41:26.377529+0000","last_active":"2026-02-25T08:41:31.507032+0000","last_peered":"2026-02-25T08:41:31.507032+0000","last_clean":"2026-02-25T08:41:31.507032+0000","last_became_active":"2026-02-25T08:41:24.426559+0000","last_became_peered":"2026-02-25T08:41:24.426559+0000","last_unstale":"2026-02-25T08:41:31.507032+0000","last_undegraded":"2026-02-25T08:41:31.507032+0000","last_fullsized":"2026-02-25T08:41:31.507032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214748299264,"available":128843481088,"internally_reserved":0,"allocated":85904818176,"data_stored":85904818176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96632737792,"internally_reserved":0,"allocated":64428486656,"data_stored":64428486656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20973260,"kb_used_data":20973260,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455524,"statfs":{"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972440,"kb_used_data":20972440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456344,"statfs":{"total":53687074816,"available":32211296256,"internally_reserved":0,"allocated":21475778560,"data_stored":21475778560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972640,"kb_used_data":20972640,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456144,"statfs":{"total":53687074816,"available":32211091456,"internally_reserved":0,"allocated":21475983360,"data_stored":21475983360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972980,"kb_used_data":20972980,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455804,"statfs":{"total":53687074816,"available":32210743296,"internally_reserved":0,"allocated":21476331520,"data_stored":21476331520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32210984960,"internally_reserved":0,"allocated":21476089856,"data_stored":21476089856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32211296256,"internally_reserved":0,"allocated":21475778560,"data_stored":21475778560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53687074816,"available":32210743296,"internally_reserved":0,"allocated":21476331520,"data_stored":21476331520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53687074816,"available":32210984960,"internally_reserved":0,"allocated":21476089856,"data_stored":21476089856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53687074816,"available":32211296256,"internally_reserved":0,"allocated":21475778560,"data_stored":21475778560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:33.761 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:33.805 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:33.803+0000 7f1cc085a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:33.808 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:33.807+0000 7f1cc085a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:33.875 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:33.871+0000 7fade5235640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:33.877 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:33.876+0000 7fade5235640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:33.988 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:33.989 INFO:teuthology.orchestra.run.trial169.stderr:dumped all 2026-02-25T08:41:33.998 INFO:teuthology.orchestra.run.trial169.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2026-02-25T08:41:33.499834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83891320,"kb_used_data":83891320,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125823816,"statfs":{"total":214748299264,"available":128843587584,"internally_reserved":0,"allocated":85904711680,"data_stored":85904711680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.173910"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.580852+0000","last_change":"2026-02-25T08:41:31.582015+0000","last_active":"2026-02-25T08:41:31.580852+0000","last_peered":"2026-02-25T08:41:31.580852+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.577339+0000","last_became_peered":"2026-02-25T08:41:31.577339+0000","last_unstale":"2026-02-25T08:41:31.580852+0000","last_undegraded":"2026-02-25T08:41:31.580852+0000","last_fullsized":"2026-02-25T08:41:31.580852+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.582809+0000","last_change":"2026-02-25T08:41:31.583321+0000","last_active":"2026-02-25T08:41:31.582809+0000","last_peered":"2026-02-25T08:41:31.582809+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.567579+0000","last_became_peered":"2026-02-25T08:41:31.567579+0000","last_unstale":"2026-02-25T08:41:31.582809+0000","last_undegraded":"2026-02-25T08:41:31.582809+0000","last_fullsized":"2026-02-25T08:41:31.582809+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.568542+0000","last_change":"2026-02-25T08:41:31.569490+0000","last_active":"2026-02-25T08:41:31.568542+0000","last_peered":"2026-02-25T08:41:31.568542+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.550562+0000","last_became_peered":"2026-02-25T08:41:31.550562+0000","last_unstale":"2026-02-25T08:41:31.568542+0000","last_undegraded":"2026-02-25T08:41:31.568542+0000","last_fullsized":"2026-02-25T08:41:31.568542+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.583197+0000","last_change":"2026-02-25T08:41:31.583740+0000","last_active":"2026-02-25T08:41:31.583197+0000","last_peered":"2026-02-25T08:41:31.583197+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.571602+0000","last_became_peered":"2026-02-25T08:41:31.571602+0000","last_unstale":"2026-02-25T08:41:31.583197+0000","last_undegraded":"2026-02-25T08:41:31.583197+0000","last_fullsized":"2026-02-25T08:41:31.583197+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1,"reported_epoch":16,"state":"creating+peering","last_fresh":"2026-02-25T08:41:30.522282+0000","last_change":"2026-02-25T08:41:30.522282+0000","last_active":"2026-02-25T08:41:30.484777+0000","last_peered":"2026-02-25T08:41:30.484777+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-25T08:41:30.522282+0000","last_undegraded":"2026-02-25T08:41:30.522282+0000","last_fullsized":"2026-02-25T08:41:30.522282+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.587533+0000","last_change":"2026-02-25T08:41:31.587841+0000","last_active":"2026-02-25T08:41:31.587533+0000","last_peered":"2026-02-25T08:41:31.587533+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.576273+0000","last_became_peered":"2026-02-25T08:41:31.576273+0000","last_unstale":"2026-02-25T08:41:31.587533+0000","last_undegraded":"2026-02-25T08:41:31.587533+0000","last_fullsized":"2026-02-25T08:41:31.587533+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.580842+0000","last_change":"2026-02-25T08:41:31.581547+0000","last_active":"2026-02-25T08:41:31.580842+0000","last_peered":"2026-02-25T08:41:31.580842+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.569266+0000","last_became_peered":"2026-02-25T08:41:31.569266+0000","last_unstale":"2026-02-25T08:41:31.580842+0000","last_undegraded":"2026-02-25T08:41:31.580842+0000","last_fullsized":"2026-02-25T08:41:31.580842+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1,"reported_epoch":16,"state":"creating+peering","last_fresh":"2026-02-25T08:41:30.511878+0000","last_change":"2026-02-25T08:41:30.511878+0000","last_active":"2026-02-25T08:41:30.484777+0000","last_peered":"2026-02-25T08:41:30.484777+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-25T08:41:30.511878+0000","last_undegraded":"2026-02-25T08:41:30.511878+0000","last_fullsized":"2026-02-25T08:41:30.511878+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.507232+0000","last_change":"2026-02-25T08:41:26.373872+0000","last_active":"2026-02-25T08:41:31.507232+0000","last_peered":"2026-02-25T08:41:31.507232+0000","last_clean":"2026-02-25T08:41:31.507232+0000","last_became_active":"2026-02-25T08:41:24.399215+0000","last_became_peered":"2026-02-25T08:41:24.399215+0000","last_unstale":"2026-02-25T08:41:31.507232+0000","last_undegraded":"2026-02-25T08:41:31.507232+0000","last_fullsized":"2026-02-25T08:41:31.507232+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1,"reported_epoch":16,"state":"creating+peering","last_fresh":"2026-02-25T08:41:30.501449+0000","last_change":"2026-02-25T08:41:30.501449+0000","last_active":"2026-02-25T08:41:30.484777+0000","last_peered":"2026-02-25T08:41:30.484777+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-25T08:41:30.501449+0000","last_undegraded":"2026-02-25T08:41:30.501449+0000","last_fullsized":"2026-02-25T08:41:30.501449+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.512406+0000","last_change":"2026-02-25T08:41:26.374903+0000","last_active":"2026-02-25T08:41:31.512406+0000","last_peered":"2026-02-25T08:41:31.512406+0000","last_clean":"2026-02-25T08:41:31.512406+0000","last_became_active":"2026-02-25T08:41:24.402449+0000","last_became_peered":"2026-02-25T08:41:24.402449+0000","last_unstale":"2026-02-25T08:41:31.512406+0000","last_undegraded":"2026-02-25T08:41:31.512406+0000","last_fullsized":"2026-02-25T08:41:31.512406+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.596834+0000","last_change":"2026-02-25T08:41:31.597333+0000","last_active":"2026-02-25T08:41:31.596834+0000","last_peered":"2026-02-25T08:41:31.596834+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.591938+0000","last_became_peered":"2026-02-25T08:41:31.591938+0000","last_unstale":"2026-02-25T08:41:31.596834+0000","last_undegraded":"2026-02-25T08:41:31.596834+0000","last_fullsized":"2026-02-25T08:41:31.596834+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":24,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.506042+0000","last_change":"2026-02-25T08:41:26.374022+0000","last_active":"2026-02-25T08:41:31.506042+0000","last_peered":"2026-02-25T08:41:31.506042+0000","last_clean":"2026-02-25T08:41:31.506042+0000","last_became_active":"2026-02-25T08:41:24.420798+0000","last_became_peered":"2026-02-25T08:41:24.420798+0000","last_unstale":"2026-02-25T08:41:31.506042+0000","last_undegraded":"2026-02-25T08:41:31.506042+0000","last_fullsized":"2026-02-25T08:41:31.506042+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.627251+0000","last_change":"2026-02-25T08:41:31.627868+0000","last_active":"2026-02-25T08:41:31.627251+0000","last_peered":"2026-02-25T08:41:31.627251+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.623233+0000","last_became_peered":"2026-02-25T08:41:31.623233+0000","last_unstale":"2026-02-25T08:41:31.627251+0000","last_undegraded":"2026-02-25T08:41:31.627251+0000","last_fullsized":"2026-02-25T08:41:31.627251+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.497473+0000","last_change":"2026-02-25T08:41:26.371067+0000","last_active":"2026-02-25T08:41:30.497473+0000","last_peered":"2026-02-25T08:41:30.497473+0000","last_clean":"2026-02-25T08:41:30.497473+0000","last_became_active":"2026-02-25T08:41:24.403313+0000","last_became_peered":"2026-02-25T08:41:24.403313+0000","last_unstale":"2026-02-25T08:41:30.497473+0000","last_undegraded":"2026-02-25T08:41:30.497473+0000","last_fullsized":"2026-02-25T08:41:30.497473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":88,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.506042+0000","last_change":"2026-02-25T08:41:24.427616+0000","last_active":"2026-02-25T08:41:31.506042+0000","last_peered":"2026-02-25T08:41:31.506042+0000","last_clean":"2026-02-25T08:41:31.506042+0000","last_became_active":"2026-02-25T08:41:24.406166+0000","last_became_peered":"2026-02-25T08:41:24.406166+0000","last_unstale":"2026-02-25T08:41:31.506042+0000","last_undegraded":"2026-02-25T08:41:31.506042+0000","last_fullsized":"2026-02-25T08:41:31.506042+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.567233+0000","last_change":"2026-02-25T08:41:31.567773+0000","last_active":"2026-02-25T08:41:31.567233+0000","last_peered":"2026-02-25T08:41:31.567233+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.558932+0000","last_became_peered":"2026-02-25T08:41:31.558932+0000","last_unstale":"2026-02-25T08:41:31.567233+0000","last_undegraded":"2026-02-25T08:41:31.567233+0000","last_fullsized":"2026-02-25T08:41:31.567233+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.608543+0000","last_change":"2026-02-25T08:41:31.609093+0000","last_active":"2026-02-25T08:41:31.608543+0000","last_peered":"2026-02-25T08:41:31.608543+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.602591+0000","last_became_peered":"2026-02-25T08:41:31.602591+0000","last_unstale":"2026-02-25T08:41:31.608543+0000","last_undegraded":"2026-02-25T08:41:31.608543+0000","last_fullsized":"2026-02-25T08:41:31.608543+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.498290+0000","last_change":"2026-02-25T08:41:26.371743+0000","last_active":"2026-02-25T08:41:30.498290+0000","last_peered":"2026-02-25T08:41:30.498290+0000","last_clean":"2026-02-25T08:41:30.498290+0000","last_became_active":"2026-02-25T08:41:24.408398+0000","last_became_peered":"2026-02-25T08:41:24.408398+0000","last_unstale":"2026-02-25T08:41:30.498290+0000","last_undegraded":"2026-02-25T08:41:30.498290+0000","last_fullsized":"2026-02-25T08:41:30.498290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.563878+0000","last_change":"2026-02-25T08:41:31.564506+0000","last_active":"2026-02-25T08:41:31.563878+0000","last_peered":"2026-02-25T08:41:31.563878+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.559738+0000","last_became_peered":"2026-02-25T08:41:31.559738+0000","last_unstale":"2026-02-25T08:41:31.563878+0000","last_undegraded":"2026-02-25T08:41:31.563878+0000","last_fullsized":"2026-02-25T08:41:31.563878+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.508306+0000","last_change":"2026-02-25T08:41:26.375854+0000","last_active":"2026-02-25T08:41:31.508306+0000","last_peered":"2026-02-25T08:41:31.508306+0000","last_clean":"2026-02-25T08:41:31.508306+0000","last_became_active":"2026-02-25T08:41:24.420122+0000","last_became_peered":"2026-02-25T08:41:24.420122+0000","last_unstale":"2026-02-25T08:41:31.508306+0000","last_undegraded":"2026-02-25T08:41:31.508306+0000","last_fullsized":"2026-02-25T08:41:31.508306+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.557289+0000","last_change":"2026-02-25T08:41:31.558232+0000","last_active":"2026-02-25T08:41:31.557289+0000","last_peered":"2026-02-25T08:41:31.557289+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.548939+0000","last_became_peered":"2026-02-25T08:41:31.548939+0000","last_unstale":"2026-02-25T08:41:31.557289+0000","last_undegraded":"2026-02-25T08:41:31.557289+0000","last_fullsized":"2026-02-25T08:41:31.557289+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.507371+0000","last_change":"2026-02-25T08:41:26.377253+0000","last_active":"2026-02-25T08:41:31.507371+0000","last_peered":"2026-02-25T08:41:31.507371+0000","last_clean":"2026-02-25T08:41:31.507371+0000","last_became_active":"2026-02-25T08:41:24.409565+0000","last_became_peered":"2026-02-25T08:41:24.409565+0000","last_unstale":"2026-02-25T08:41:31.507371+0000","last_undegraded":"2026-02-25T08:41:31.507371+0000","last_fullsized":"2026-02-25T08:41:31.507371+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.592738+0000","last_change":"2026-02-25T08:41:31.593574+0000","last_active":"2026-02-25T08:41:31.592738+0000","last_peered":"2026-02-25T08:41:31.592738+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.586165+0000","last_became_peered":"2026-02-25T08:41:31.586165+0000","last_unstale":"2026-02-25T08:41:31.592738+0000","last_undegraded":"2026-02-25T08:41:31.592738+0000","last_fullsized":"2026-02-25T08:41:31.592738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.507032+0000","last_change":"2026-02-25T08:41:26.377529+0000","last_active":"2026-02-25T08:41:31.507032+0000","last_peered":"2026-02-25T08:41:31.507032+0000","last_clean":"2026-02-25T08:41:31.507032+0000","last_became_active":"2026-02-25T08:41:24.426559+0000","last_became_peered":"2026-02-25T08:41:24.426559+0000","last_unstale":"2026-02-25T08:41:31.507032+0000","last_undegraded":"2026-02-25T08:41:31.507032+0000","last_fullsized":"2026-02-25T08:41:31.507032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214748299264,"available":128843481088,"internally_reserved":0,"allocated":85904818176,"data_stored":85904818176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96632737792,"internally_reserved":0,"allocated":64428486656,"data_stored":64428486656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20973260,"kb_used_data":20973260,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455524,"statfs":{"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972440,"kb_used_data":20972440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456344,"statfs":{"total":53687074816,"available":32211296256,"internally_reserved":0,"allocated":21475778560,"data_stored":21475778560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972640,"kb_used_data":20972640,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456144,"statfs":{"total":53687074816,"available":32211091456,"internally_reserved":0,"allocated":21475983360,"data_stored":21475983360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972980,"kb_used_data":20972980,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455804,"statfs":{"total":53687074816,"available":32210743296,"internally_reserved":0,"allocated":21476331520,"data_stored":21476331520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32210984960,"internally_reserved":0,"allocated":21476089856,"data_stored":21476089856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32211296256,"internally_reserved":0,"allocated":21475778560,"data_stored":21475778560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53687074816,"available":32210743296,"internally_reserved":0,"allocated":21476331520,"data_stored":21476331520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53687074816,"available":32210984960,"internally_reserved":0,"allocated":21476089856,"data_stored":21476089856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53687074816,"available":32211296256,"internally_reserved":0,"allocated":21475778560,"data_stored":21475778560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:34.000 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-25T08:41:34.113 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:34.110+0000 7fd86bacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:34.116 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:34.114+0000 7fd86bacb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:34.267 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:34.268 INFO:teuthology.orchestra.run.trial169.stdout:{"fsid":"a3da792f-4239-4b1a-ad7d-525e6c05ee6a","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":21,"monmap":{"epoch":1,"min_mon_release_name":"tentacle","num_mons":1},"osdmap":{"epoch":19,"num_osds":4,"num_up_osds":4,"osd_up_since":1772008882,"num_in_osds":4,"osd_in_since":1772008874,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":22},{"state_name":"creating+peering","count":3}],"num_pgs":25,"num_pools":3,"num_objects":4,"data_bytes":590387,"bytes_used":85904711680,"bytes_avail":128843587584,"bytes_total":214748299264,"inactive_pgs_ratio":0.11999999731779099},"fsmap":{"epoch":1,"btime":"2026-02-25T08:41:13:109250+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs"],"services":{}},"servicemap":{"epoch":1,"modified":"2026-02-25T08:41:13.110236+0000","services":{}},"progress_events":{}} 2026-02-25T08:41:34.277 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2026-02-25T08:41:34.277 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:34.353 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:34.349+0000 7f3bb005a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:34.355 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:34.353+0000 7f3bb005a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:34.467 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:34.468 INFO:teuthology.orchestra.run.trial169.stderr:dumped all 2026-02-25T08:41:34.476 INFO:teuthology.orchestra.run.trial169.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2026-02-25T08:41:33.499834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":81,"num_write_kb":1108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55,"ondisk_log_size":55,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":83891320,"kb_used_data":83891320,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125823816,"statfs":{"total":214748299264,"available":128843587584,"internally_reserved":0,"allocated":85904711680,"data_stored":85904711680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.173910"},"pg_stats":[{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.580852+0000","last_change":"2026-02-25T08:41:31.582015+0000","last_active":"2026-02-25T08:41:31.580852+0000","last_peered":"2026-02-25T08:41:31.580852+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.577339+0000","last_became_peered":"2026-02-25T08:41:31.577339+0000","last_unstale":"2026-02-25T08:41:31.580852+0000","last_undegraded":"2026-02-25T08:41:31.580852+0000","last_fullsized":"2026-02-25T08:41:31.580852+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.582809+0000","last_change":"2026-02-25T08:41:31.583321+0000","last_active":"2026-02-25T08:41:31.582809+0000","last_peered":"2026-02-25T08:41:31.582809+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.567579+0000","last_became_peered":"2026-02-25T08:41:31.567579+0000","last_unstale":"2026-02-25T08:41:31.582809+0000","last_undegraded":"2026-02-25T08:41:31.582809+0000","last_fullsized":"2026-02-25T08:41:31.582809+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.568542+0000","last_change":"2026-02-25T08:41:31.569490+0000","last_active":"2026-02-25T08:41:31.568542+0000","last_peered":"2026-02-25T08:41:31.568542+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.550562+0000","last_became_peered":"2026-02-25T08:41:31.550562+0000","last_unstale":"2026-02-25T08:41:31.568542+0000","last_undegraded":"2026-02-25T08:41:31.568542+0000","last_fullsized":"2026-02-25T08:41:31.568542+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.583197+0000","last_change":"2026-02-25T08:41:31.583740+0000","last_active":"2026-02-25T08:41:31.583197+0000","last_peered":"2026-02-25T08:41:31.583197+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.571602+0000","last_became_peered":"2026-02-25T08:41:31.571602+0000","last_unstale":"2026-02-25T08:41:31.583197+0000","last_undegraded":"2026-02-25T08:41:31.583197+0000","last_fullsized":"2026-02-25T08:41:31.583197+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1,"reported_epoch":16,"state":"creating+peering","last_fresh":"2026-02-25T08:41:30.522282+0000","last_change":"2026-02-25T08:41:30.522282+0000","last_active":"2026-02-25T08:41:30.484777+0000","last_peered":"2026-02-25T08:41:30.484777+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-25T08:41:30.522282+0000","last_undegraded":"2026-02-25T08:41:30.522282+0000","last_fullsized":"2026-02-25T08:41:30.522282+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.587533+0000","last_change":"2026-02-25T08:41:31.587841+0000","last_active":"2026-02-25T08:41:31.587533+0000","last_peered":"2026-02-25T08:41:31.587533+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.576273+0000","last_became_peered":"2026-02-25T08:41:31.576273+0000","last_unstale":"2026-02-25T08:41:31.587533+0000","last_undegraded":"2026-02-25T08:41:31.587533+0000","last_fullsized":"2026-02-25T08:41:31.587533+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.580842+0000","last_change":"2026-02-25T08:41:31.581547+0000","last_active":"2026-02-25T08:41:31.580842+0000","last_peered":"2026-02-25T08:41:31.580842+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.569266+0000","last_became_peered":"2026-02-25T08:41:31.569266+0000","last_unstale":"2026-02-25T08:41:31.580842+0000","last_undegraded":"2026-02-25T08:41:31.580842+0000","last_fullsized":"2026-02-25T08:41:31.580842+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1,"reported_epoch":16,"state":"creating+peering","last_fresh":"2026-02-25T08:41:30.511878+0000","last_change":"2026-02-25T08:41:30.511878+0000","last_active":"2026-02-25T08:41:30.484777+0000","last_peered":"2026-02-25T08:41:30.484777+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-25T08:41:30.511878+0000","last_undegraded":"2026-02-25T08:41:30.511878+0000","last_fullsized":"2026-02-25T08:41:30.511878+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.507232+0000","last_change":"2026-02-25T08:41:26.373872+0000","last_active":"2026-02-25T08:41:31.507232+0000","last_peered":"2026-02-25T08:41:31.507232+0000","last_clean":"2026-02-25T08:41:31.507232+0000","last_became_active":"2026-02-25T08:41:24.399215+0000","last_became_peered":"2026-02-25T08:41:24.399215+0000","last_unstale":"2026-02-25T08:41:31.507232+0000","last_undegraded":"2026-02-25T08:41:31.507232+0000","last_fullsized":"2026-02-25T08:41:31.507232+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":1,"reported_epoch":16,"state":"creating+peering","last_fresh":"2026-02-25T08:41:30.501449+0000","last_change":"2026-02-25T08:41:30.501449+0000","last_active":"2026-02-25T08:41:30.484777+0000","last_peered":"2026-02-25T08:41:30.484777+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2026-02-25T08:41:30.501449+0000","last_undegraded":"2026-02-25T08:41:30.501449+0000","last_fullsized":"2026-02-25T08:41:30.501449+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.512406+0000","last_change":"2026-02-25T08:41:26.374903+0000","last_active":"2026-02-25T08:41:31.512406+0000","last_peered":"2026-02-25T08:41:31.512406+0000","last_clean":"2026-02-25T08:41:31.512406+0000","last_became_active":"2026-02-25T08:41:24.402449+0000","last_became_peered":"2026-02-25T08:41:24.402449+0000","last_unstale":"2026-02-25T08:41:31.512406+0000","last_undegraded":"2026-02-25T08:41:31.512406+0000","last_fullsized":"2026-02-25T08:41:31.512406+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.596834+0000","last_change":"2026-02-25T08:41:31.597333+0000","last_active":"2026-02-25T08:41:31.596834+0000","last_peered":"2026-02-25T08:41:31.596834+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.591938+0000","last_became_peered":"2026-02-25T08:41:31.591938+0000","last_unstale":"2026-02-25T08:41:31.596834+0000","last_undegraded":"2026-02-25T08:41:31.596834+0000","last_fullsized":"2026-02-25T08:41:31.596834+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":24,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.506042+0000","last_change":"2026-02-25T08:41:26.374022+0000","last_active":"2026-02-25T08:41:31.506042+0000","last_peered":"2026-02-25T08:41:31.506042+0000","last_clean":"2026-02-25T08:41:31.506042+0000","last_became_active":"2026-02-25T08:41:24.420798+0000","last_became_peered":"2026-02-25T08:41:24.420798+0000","last_unstale":"2026-02-25T08:41:31.506042+0000","last_undegraded":"2026-02-25T08:41:31.506042+0000","last_fullsized":"2026-02-25T08:41:31.506042+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.627251+0000","last_change":"2026-02-25T08:41:31.627868+0000","last_active":"2026-02-25T08:41:31.627251+0000","last_peered":"2026-02-25T08:41:31.627251+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.623233+0000","last_became_peered":"2026-02-25T08:41:31.623233+0000","last_unstale":"2026-02-25T08:41:31.627251+0000","last_undegraded":"2026-02-25T08:41:31.627251+0000","last_fullsized":"2026-02-25T08:41:31.627251+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":20,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.497473+0000","last_change":"2026-02-25T08:41:26.371067+0000","last_active":"2026-02-25T08:41:30.497473+0000","last_peered":"2026-02-25T08:41:30.497473+0000","last_clean":"2026-02-25T08:41:30.497473+0000","last_became_active":"2026-02-25T08:41:24.403313+0000","last_became_peered":"2026-02-25T08:41:24.403313+0000","last_unstale":"2026-02-25T08:41:30.497473+0000","last_undegraded":"2026-02-25T08:41:30.497473+0000","last_fullsized":"2026-02-25T08:41:30.497473+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":88,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.506042+0000","last_change":"2026-02-25T08:41:24.427616+0000","last_active":"2026-02-25T08:41:31.506042+0000","last_peered":"2026-02-25T08:41:31.506042+0000","last_clean":"2026-02-25T08:41:31.506042+0000","last_became_active":"2026-02-25T08:41:24.406166+0000","last_became_peered":"2026-02-25T08:41:24.406166+0000","last_unstale":"2026-02-25T08:41:31.506042+0000","last_undegraded":"2026-02-25T08:41:31.506042+0000","last_fullsized":"2026-02-25T08:41:31.506042+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.567233+0000","last_change":"2026-02-25T08:41:31.567773+0000","last_active":"2026-02-25T08:41:31.567233+0000","last_peered":"2026-02-25T08:41:31.567233+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.558932+0000","last_became_peered":"2026-02-25T08:41:31.558932+0000","last_unstale":"2026-02-25T08:41:31.567233+0000","last_undegraded":"2026-02-25T08:41:31.567233+0000","last_fullsized":"2026-02-25T08:41:31.567233+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.608543+0000","last_change":"2026-02-25T08:41:31.609093+0000","last_active":"2026-02-25T08:41:31.608543+0000","last_peered":"2026-02-25T08:41:31.608543+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.602591+0000","last_became_peered":"2026-02-25T08:41:31.602591+0000","last_unstale":"2026-02-25T08:41:31.608543+0000","last_undegraded":"2026-02-25T08:41:31.608543+0000","last_fullsized":"2026-02-25T08:41:31.608543+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"active+clean","last_fresh":"2026-02-25T08:41:30.498290+0000","last_change":"2026-02-25T08:41:26.371743+0000","last_active":"2026-02-25T08:41:30.498290+0000","last_peered":"2026-02-25T08:41:30.498290+0000","last_clean":"2026-02-25T08:41:30.498290+0000","last_became_active":"2026-02-25T08:41:24.408398+0000","last_became_peered":"2026-02-25T08:41:24.408398+0000","last_unstale":"2026-02-25T08:41:30.498290+0000","last_undegraded":"2026-02-25T08:41:30.498290+0000","last_fullsized":"2026-02-25T08:41:30.498290+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.563878+0000","last_change":"2026-02-25T08:41:31.564506+0000","last_active":"2026-02-25T08:41:31.563878+0000","last_peered":"2026-02-25T08:41:31.563878+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.559738+0000","last_became_peered":"2026-02-25T08:41:31.559738+0000","last_unstale":"2026-02-25T08:41:31.563878+0000","last_undegraded":"2026-02-25T08:41:31.563878+0000","last_fullsized":"2026-02-25T08:41:31.563878+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.508306+0000","last_change":"2026-02-25T08:41:26.375854+0000","last_active":"2026-02-25T08:41:31.508306+0000","last_peered":"2026-02-25T08:41:31.508306+0000","last_clean":"2026-02-25T08:41:31.508306+0000","last_became_active":"2026-02-25T08:41:24.420122+0000","last_became_peered":"2026-02-25T08:41:24.420122+0000","last_unstale":"2026-02-25T08:41:31.508306+0000","last_undegraded":"2026-02-25T08:41:31.508306+0000","last_fullsized":"2026-02-25T08:41:31.508306+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.557289+0000","last_change":"2026-02-25T08:41:31.558232+0000","last_active":"2026-02-25T08:41:31.557289+0000","last_peered":"2026-02-25T08:41:31.557289+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.548939+0000","last_became_peered":"2026-02-25T08:41:31.548939+0000","last_unstale":"2026-02-25T08:41:31.557289+0000","last_undegraded":"2026-02-25T08:41:31.557289+0000","last_fullsized":"2026-02-25T08:41:31.557289+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.507371+0000","last_change":"2026-02-25T08:41:26.377253+0000","last_active":"2026-02-25T08:41:31.507371+0000","last_peered":"2026-02-25T08:41:31.507371+0000","last_clean":"2026-02-25T08:41:31.507371+0000","last_became_active":"2026-02-25T08:41:24.409565+0000","last_became_peered":"2026-02-25T08:41:24.409565+0000","last_unstale":"2026-02-25T08:41:31.507371+0000","last_undegraded":"2026-02-25T08:41:31.507371+0000","last_fullsized":"2026-02-25T08:41:31.507371+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.592738+0000","last_change":"2026-02-25T08:41:31.593574+0000","last_active":"2026-02-25T08:41:31.592738+0000","last_peered":"2026-02-25T08:41:31.592738+0000","last_clean":"2026-02-25T08:41:30.484777+0000","last_became_active":"2026-02-25T08:41:31.586165+0000","last_became_peered":"2026-02-25T08:41:31.586165+0000","last_unstale":"2026-02-25T08:41:31.592738+0000","last_undegraded":"2026-02-25T08:41:31.592738+0000","last_fullsized":"2026-02-25T08:41:31.592738+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2026-02-25T08:41:31.507032+0000","last_change":"2026-02-25T08:41:26.377529+0000","last_active":"2026-02-25T08:41:31.507032+0000","last_peered":"2026-02-25T08:41:31.507032+0000","last_clean":"2026-02-25T08:41:31.507032+0000","last_became_active":"2026-02-25T08:41:24.426559+0000","last_became_peered":"2026-02-25T08:41:24.426559+0000","last_unstale":"2026-02-25T08:41:31.507032+0000","last_undegraded":"2026-02-25T08:41:31.507032+0000","last_fullsized":"2026-02-25T08:41:31.507032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214748299264,"available":128843481088,"internally_reserved":0,"allocated":85904818176,"data_stored":85904818176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96632737792,"internally_reserved":0,"allocated":64428486656,"data_stored":64428486656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20973260,"kb_used_data":20973260,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455524,"statfs":{"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972440,"kb_used_data":20972440,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456344,"statfs":{"total":53687074816,"available":32211296256,"internally_reserved":0,"allocated":21475778560,"data_stored":21475778560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672969,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972640,"kb_used_data":20972640,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31456144,"statfs":{"total":53687074816,"available":32211091456,"internally_reserved":0,"allocated":21475983360,"data_stored":21475983360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640266,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":20972980,"kb_used_data":20972980,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31455804,"statfs":{"total":53687074816,"available":32210743296,"internally_reserved":0,"allocated":21476331520,"data_stored":21476331520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32210984960,"internally_reserved":0,"allocated":21476089856,"data_stored":21476089856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32211296256,"internally_reserved":0,"allocated":21475778560,"data_stored":21475778560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53687074816,"available":32210743296,"internally_reserved":0,"allocated":21476331520,"data_stored":21476331520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53687074816,"available":32210984960,"internally_reserved":0,"allocated":21476089856,"data_stored":21476089856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53687074816,"available":32211296256,"internally_reserved":0,"allocated":21475778560,"data_stored":21475778560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53687074816,"available":32210456576,"internally_reserved":0,"allocated":21476618240,"data_stored":21476618240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:34.501 INFO:tasks.rados.rados.0.trial169.stdout:1: write initial oid 1 2026-02-25T08:41:34.501 INFO:tasks.rados.rados.0.trial169.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2026-02-25T08:41:34.504 INFO:tasks.rados.rados.0.trial169.stdout:1: writing trial16941936-1 from 416807 to 892056 tid 1 2026-02-25T08:41:34.509 INFO:tasks.rados.rados.0.trial169.stdout:1: writing trial16941936-1 from 1542129 to 2016807 tid 2 2026-02-25T08:41:34.509 INFO:tasks.rados.rados.0.trial169.stdout:2: write initial oid 2 2026-02-25T08:41:34.509 INFO:tasks.rados.rados.0.trial169.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2026-02-25T08:41:34.513 INFO:tasks.rados.rados.0.trial169.stdout:2: writing trial16941936-2 from 416807 to 892056 tid 1 2026-02-25T08:41:34.518 INFO:tasks.rados.rados.0.trial169.stdout:2: writing trial16941936-2 from 1542129 to 2016807 tid 2 2026-02-25T08:41:34.518 INFO:tasks.rados.rados.0.trial169.stdout:3: write initial oid 3 2026-02-25T08:41:34.518 INFO:tasks.rados.rados.0.trial169.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2026-02-25T08:41:34.521 INFO:tasks.rados.rados.0.trial169.stdout:3: writing trial16941936-3 from 433614 to 984112 tid 1 2026-02-25T08:41:34.525 INFO:tasks.rados.rados.0.trial169.stdout:3: writing trial16941936-3 from 1600611 to 2033614 tid 2 2026-02-25T08:41:34.525 INFO:tasks.rados.rados.0.trial169.stdout:4: write initial oid 4 2026-02-25T08:41:34.526 INFO:tasks.rados.rados.0.trial169.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2026-02-25T08:41:34.530 INFO:tasks.rados.rados.0.trial169.stdout:4: writing trial16941936-4 from 450421 to 1076168 tid 1 2026-02-25T08:41:34.534 INFO:tasks.rados.rados.0.trial169.stdout:4: writing trial16941936-4 from 1659093 to 2050421 tid 2 2026-02-25T08:41:34.534 INFO:tasks.rados.rados.0.trial169.stdout:5: write initial oid 5 2026-02-25T08:41:34.534 INFO:tasks.rados.rados.0.trial169.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2026-02-25T08:41:34.538 INFO:tasks.rados.rados.0.trial169.stdout:5: writing trial16941936-5 from 467228 to 1168224 tid 1 2026-02-25T08:41:34.542 INFO:tasks.rados.rados.0.trial169.stdout:5: writing trial16941936-5 from 1717575 to 2067228 tid 2 2026-02-25T08:41:34.542 INFO:tasks.rados.rados.0.trial169.stdout:6: write initial oid 6 2026-02-25T08:41:34.542 INFO:tasks.rados.rados.0.trial169.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2026-02-25T08:41:34.547 INFO:tasks.rados.rados.0.trial169.stdout:6: writing trial16941936-6 from 484035 to 1260280 tid 1 2026-02-25T08:41:34.550 INFO:tasks.rados.rados.0.trial169.stdout:6: writing trial16941936-6 from 2059704 to 2084035 tid 2 2026-02-25T08:41:34.550 INFO:tasks.rados.rados.0.trial169.stdout:7: write initial oid 7 2026-02-25T08:41:34.550 INFO:tasks.rados.rados.0.trial169.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2026-02-25T08:41:34.553 INFO:tasks.rados.rados.0.trial169.stdout:7: writing trial16941936-7 from 500842 to 952336 tid 1 2026-02-25T08:41:34.558 INFO:tasks.rados.rados.0.trial169.stdout:7: writing trial16941936-7 from 1718186 to 2100842 tid 2 2026-02-25T08:41:34.558 INFO:tasks.rados.rados.0.trial169.stdout:8: write initial oid 8 2026-02-25T08:41:34.558 INFO:tasks.rados.rados.0.trial169.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2026-02-25T08:41:34.562 INFO:tasks.rados.rados.0.trial169.stdout:8: writing trial16941936-8 from 517649 to 1044392 tid 1 2026-02-25T08:41:34.566 INFO:tasks.rados.rados.0.trial169.stdout:8: writing trial16941936-8 from 1776668 to 2117649 tid 2 2026-02-25T08:41:34.566 INFO:tasks.rados.rados.0.trial169.stdout:9: write initial oid 9 2026-02-25T08:41:34.567 INFO:tasks.rados.rados.0.trial169.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2026-02-25T08:41:34.571 INFO:tasks.rados.rados.0.trial169.stdout:9: writing trial16941936-9 from 534456 to 1252801 tid 1 2026-02-25T08:41:34.575 INFO:tasks.rados.rados.0.trial169.stdout:9: writing trial16941936-9 from 1951503 to 2134456 tid 2 2026-02-25T08:41:34.575 INFO:tasks.rados.rados.0.trial169.stdout:10: write initial oid 10 2026-02-25T08:41:34.575 INFO:tasks.rados.rados.0.trial169.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2026-02-25T08:41:34.579 INFO:tasks.rados.rados.0.trial169.stdout:10: writing trial16941936-10 from 551263 to 1344857 tid 1 2026-02-25T08:41:34.583 INFO:tasks.rados.rados.0.trial169.stdout:10: writing trial16941936-10 from 1893632 to 2151263 tid 2 2026-02-25T08:41:34.583 INFO:tasks.rados.rados.0.trial169.stdout:11: write initial oid 11 2026-02-25T08:41:34.583 INFO:tasks.rados.rados.0.trial169.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2026-02-25T08:41:34.587 INFO:tasks.rados.rados.0.trial169.stdout:11: writing trial16941936-11 from 568070 to 1036913 tid 1 2026-02-25T08:41:34.591 INFO:tasks.rados.rados.0.trial169.stdout:11: writing trial16941936-11 from 1552114 to 2168070 tid 2 2026-02-25T08:41:34.591 INFO:tasks.rados.rados.0.trial169.stdout:12: write initial oid 12 2026-02-25T08:41:34.591 INFO:tasks.rados.rados.0.trial169.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2026-02-25T08:41:34.595 INFO:tasks.rados.rados.0.trial169.stdout:12: writing trial16941936-12 from 584877 to 1128969 tid 1 2026-02-25T08:41:34.600 INFO:tasks.rados.rados.0.trial169.stdout:12: writing trial16941936-12 from 1610596 to 2172599 tid 2 2026-02-25T08:41:34.600 INFO:tasks.rados.rados.0.trial169.stdout:13: write initial oid 13 2026-02-25T08:41:34.600 INFO:tasks.rados.rados.0.trial169.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2026-02-25T08:41:34.604 INFO:tasks.rados.rados.0.trial169.stdout:13: writing trial16941936-13 from 601684 to 1221025 tid 1 2026-02-25T08:41:34.608 INFO:tasks.rados.rados.0.trial169.stdout:13: writing trial16941936-13 from 1669078 to 2201684 tid 2 2026-02-25T08:41:34.608 INFO:tasks.rados.rados.0.trial169.stdout:14: write initial oid 14 2026-02-25T08:41:34.609 INFO:tasks.rados.rados.0.trial169.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2026-02-25T08:41:34.614 INFO:tasks.rados.rados.0.trial169.stdout:14: writing trial16941936-14 from 618491 to 1313081 tid 1 2026-02-25T08:41:34.617 INFO:tasks.rados.rados.0.trial169.stdout:14: writing trial16941936-14 from 2011207 to 2218491 tid 2 2026-02-25T08:41:34.617 INFO:tasks.rados.rados.0.trial169.stdout:15: write initial oid 15 2026-02-25T08:41:34.617 INFO:tasks.rados.rados.0.trial169.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2026-02-25T08:41:34.622 INFO:tasks.rados.rados.0.trial169.stdout:15: writing trial16941936-15 from 635298 to 1405137 tid 1 2026-02-25T08:41:34.626 INFO:tasks.rados.rados.0.trial169.stdout:15: writing trial16941936-15 from 2069689 to 2235298 tid 2 2026-02-25T08:41:34.626 INFO:tasks.rados.rados.0.trial169.stdout:16: write initial oid 16 2026-02-25T08:41:34.626 INFO:tasks.rados.rados.0.trial169.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2026-02-25T08:41:34.629 INFO:tasks.rados.rados.0.trial169.stdout:16: writing trial16941936-16 from 652105 to 1097193 tid 1 2026-02-25T08:41:34.634 INFO:tasks.rados.rados.0.trial169.stdout:16: writing trial16941936-16 from 1728171 to 2181159 tid 2 2026-02-25T08:41:34.635 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.635 INFO:tasks.rados.rados.0.trial169.stdout:2: finishing write tid 1 to trial16941936-2 2026-02-25T08:41:34.635 INFO:tasks.rados.rados.0.trial169.stdout:1: finishing write tid 1 to trial16941936-1 2026-02-25T08:41:34.635 INFO:tasks.rados.rados.0.trial169.stdout:2: finishing write tid 2 to trial16941936-2 2026-02-25T08:41:34.635 INFO:tasks.rados.rados.0.trial169.stdout:1: finishing write tid 2 to trial16941936-1 2026-02-25T08:41:34.635 INFO:tasks.rados.rados.0.trial169.stdout:2: finishing write tid 3 to trial16941936-2 2026-02-25T08:41:34.635 INFO:tasks.rados.rados.0.trial169.stdout:2: finishing write tid 4 to trial16941936-2 2026-02-25T08:41:34.635 INFO:tasks.rados.rados.0.trial169.stdout:2: oid 2 updating version 0 to 1 2026-02-25T08:41:34.635 INFO:tasks.rados.rados.0.trial169.stdout:2: oid 2 updating version 1 to 3 2026-02-25T08:41:34.635 INFO:tasks.rados.rados.0.trial169.stdout:2: oid 2 version 3 is already newer than 3 2026-02-25T08:41:34.635 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2026-02-25T08:41:34.636 INFO:tasks.rados.rados.0.trial169.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2026-02-25T08:41:34.636 INFO:tasks.rados.rados.0.trial169.stdout:1: finishing write tid 3 to trial16941936-1 2026-02-25T08:41:34.636 INFO:tasks.rados.rados.0.trial169.stdout:1: finishing write tid 4 to trial16941936-1 2026-02-25T08:41:34.636 INFO:tasks.rados.rados.0.trial169.stdout:1: oid 1 updating version 0 to 3 2026-02-25T08:41:34.636 INFO:tasks.rados.rados.0.trial169.stdout:1: oid 1 version 3 is already newer than 3 2026-02-25T08:41:34.636 INFO:tasks.rados.rados.0.trial169.stdout:1: oid 1 version 3 is already newer than 1 2026-02-25T08:41:34.636 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2026-02-25T08:41:34.636 INFO:tasks.rados.rados.0.trial169.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2026-02-25T08:41:34.636 INFO:tasks.rados.rados.0.trial169.stdout:3: finishing write tid 1 to trial16941936-3 2026-02-25T08:41:34.636 INFO:tasks.rados.rados.0.trial169.stdout:1: done (15 left) 2026-02-25T08:41:34.636 INFO:tasks.rados.rados.0.trial169.stdout:2: done (14 left) 2026-02-25T08:41:34.636 INFO:tasks.rados.rados.0.trial169.stdout:17: write initial oid 17 2026-02-25T08:41:34.637 INFO:tasks.rados.rados.0.trial169.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2026-02-25T08:41:34.639 INFO:tasks.rados.rados.0.trial169.stdout:17: writing trial16941936-17 from 668912 to 1305602 tid 1 2026-02-25T08:41:34.643 INFO:tasks.rados.rados.0.trial169.stdout:17: writing trial16941936-17 from 1903006 to 2268912 tid 2 2026-02-25T08:41:34.643 INFO:tasks.rados.rados.0.trial169.stdout:18: write initial oid 18 2026-02-25T08:41:34.643 INFO:tasks.rados.rados.0.trial169.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2026-02-25T08:41:34.648 INFO:tasks.rados.rados.0.trial169.stdout:18: writing trial16941936-18 from 685719 to 1397658 tid 1 2026-02-25T08:41:34.653 INFO:tasks.rados.rados.0.trial169.stdout:18: writing trial16941936-18 from 1845135 to 2285719 tid 2 2026-02-25T08:41:34.653 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.653 INFO:tasks.rados.rados.0.trial169.stdout:4: finishing write tid 1 to trial16941936-4 2026-02-25T08:41:34.653 INFO:tasks.rados.rados.0.trial169.stdout:8: finishing write tid 1 to trial16941936-8 2026-02-25T08:41:34.653 INFO:tasks.rados.rados.0.trial169.stdout:6: finishing write tid 1 to trial16941936-6 2026-02-25T08:41:34.653 INFO:tasks.rados.rados.0.trial169.stdout:4: finishing write tid 2 to trial16941936-4 2026-02-25T08:41:34.653 INFO:tasks.rados.rados.0.trial169.stdout:9: finishing write tid 1 to trial16941936-9 2026-02-25T08:41:34.654 INFO:tasks.rados.rados.0.trial169.stdout:6: finishing write tid 2 to trial16941936-6 2026-02-25T08:41:34.654 INFO:tasks.rados.rados.0.trial169.stdout:4: finishing write tid 3 to trial16941936-4 2026-02-25T08:41:34.654 INFO:tasks.rados.rados.0.trial169.stdout:4: finishing write tid 4 to trial16941936-4 2026-02-25T08:41:34.654 INFO:tasks.rados.rados.0.trial169.stdout:4: oid 4 updating version 0 to 1 2026-02-25T08:41:34.654 INFO:tasks.rados.rados.0.trial169.stdout:4: oid 4 updating version 1 to 3 2026-02-25T08:41:34.654 INFO:tasks.rados.rados.0.trial169.stdout:4: oid 4 version 3 is already newer than 3 2026-02-25T08:41:34.654 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2026-02-25T08:41:34.654 INFO:tasks.rados.rados.0.trial169.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2026-02-25T08:41:34.654 INFO:tasks.rados.rados.0.trial169.stdout:6: finishing write tid 3 to trial16941936-6 2026-02-25T08:41:34.654 INFO:tasks.rados.rados.0.trial169.stdout:8: finishing write tid 2 to trial16941936-8 2026-02-25T08:41:34.654 INFO:tasks.rados.rados.0.trial169.stdout:6: finishing write tid 4 to trial16941936-6 2026-02-25T08:41:34.654 INFO:tasks.rados.rados.0.trial169.stdout:6: oid 6 updating version 0 to 1 2026-02-25T08:41:34.655 INFO:tasks.rados.rados.0.trial169.stdout:6: oid 6 updating version 1 to 3 2026-02-25T08:41:34.655 INFO:tasks.rados.rados.0.trial169.stdout:6: oid 6 version 3 is already newer than 3 2026-02-25T08:41:34.655 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2026-02-25T08:41:34.655 INFO:tasks.rados.rados.0.trial169.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2026-02-25T08:41:34.655 INFO:tasks.rados.rados.0.trial169.stdout:7: finishing write tid 1 to trial16941936-7 2026-02-25T08:41:34.655 INFO:tasks.rados.rados.0.trial169.stdout:11: finishing write tid 1 to trial16941936-11 2026-02-25T08:41:34.655 INFO:tasks.rados.rados.0.trial169.stdout:9: finishing write tid 2 to trial16941936-9 2026-02-25T08:41:34.655 INFO:tasks.rados.rados.0.trial169.stdout:8: finishing write tid 3 to trial16941936-8 2026-02-25T08:41:34.655 INFO:tasks.rados.rados.0.trial169.stdout:4: done (15 left) 2026-02-25T08:41:34.655 INFO:tasks.rados.rados.0.trial169.stdout:6: done (14 left) 2026-02-25T08:41:34.655 INFO:tasks.rados.rados.0.trial169.stdout:19: write initial oid 19 2026-02-25T08:41:34.655 INFO:tasks.rados.rados.0.trial169.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2026-02-25T08:41:34.657 INFO:tasks.rados.rados.0.trial169.stdout:19: writing trial16941936-19 from 702526 to 1489714 tid 1 2026-02-25T08:41:34.661 INFO:tasks.rados.rados.0.trial169.stdout:19: writing trial16941936-19 from 1903617 to 2302526 tid 2 2026-02-25T08:41:34.661 INFO:tasks.rados.rados.0.trial169.stdout:20: write initial oid 20 2026-02-25T08:41:34.661 INFO:tasks.rados.rados.0.trial169.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2026-02-25T08:41:34.665 INFO:tasks.rados.rados.0.trial169.stdout:20: writing trial16941936-20 from 719333 to 1181770 tid 1 2026-02-25T08:41:34.670 INFO:tasks.rados.rados.0.trial169.stdout:20: writing trial16941936-20 from 1962099 to 2319333 tid 2 2026-02-25T08:41:34.670 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.670 INFO:tasks.rados.rados.0.trial169.stdout:9: finishing write tid 3 to trial16941936-9 2026-02-25T08:41:34.670 INFO:tasks.rados.rados.0.trial169.stdout:9: finishing write tid 4 to trial16941936-9 2026-02-25T08:41:34.671 INFO:tasks.rados.rados.0.trial169.stdout:9: oid 9 updating version 0 to 1 2026-02-25T08:41:34.671 INFO:tasks.rados.rados.0.trial169.stdout:9: oid 9 updating version 1 to 3 2026-02-25T08:41:34.671 INFO:tasks.rados.rados.0.trial169.stdout:9: oid 9 version 3 is already newer than 3 2026-02-25T08:41:34.671 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2026-02-25T08:41:34.671 INFO:tasks.rados.rados.0.trial169.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2026-02-25T08:41:34.671 INFO:tasks.rados.rados.0.trial169.stdout:12: finishing write tid 1 to trial16941936-12 2026-02-25T08:41:34.671 INFO:tasks.rados.rados.0.trial169.stdout:8: finishing write tid 4 to trial16941936-8 2026-02-25T08:41:34.671 INFO:tasks.rados.rados.0.trial169.stdout:8: oid 8 updating version 0 to 3 2026-02-25T08:41:34.671 INFO:tasks.rados.rados.0.trial169.stdout:8: oid 8 version 3 is already newer than 1 2026-02-25T08:41:34.671 INFO:tasks.rados.rados.0.trial169.stdout:8: oid 8 version 3 is already newer than 3 2026-02-25T08:41:34.671 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2026-02-25T08:41:34.671 INFO:tasks.rados.rados.0.trial169.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2026-02-25T08:41:34.672 INFO:tasks.rados.rados.0.trial169.stdout:3: finishing write tid 2 to trial16941936-3 2026-02-25T08:41:34.672 INFO:tasks.rados.rados.0.trial169.stdout:13: finishing write tid 1 to trial16941936-13 2026-02-25T08:41:34.672 INFO:tasks.rados.rados.0.trial169.stdout:10: finishing write tid 1 to trial16941936-10 2026-02-25T08:41:34.672 INFO:tasks.rados.rados.0.trial169.stdout:8: done (15 left) 2026-02-25T08:41:34.672 INFO:tasks.rados.rados.0.trial169.stdout:9: done (14 left) 2026-02-25T08:41:34.672 INFO:tasks.rados.rados.0.trial169.stdout:21: write initial oid 21 2026-02-25T08:41:34.672 INFO:tasks.rados.rados.0.trial169.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2026-02-25T08:41:34.674 INFO:tasks.rados.rados.0.trial169.stdout:21: writing trial16941936-21 from 736140 to 1273826 tid 1 2026-02-25T08:41:34.679 INFO:tasks.rados.rados.0.trial169.stdout:21: writing trial16941936-21 from 2020581 to 2336140 tid 2 2026-02-25T08:41:34.679 INFO:tasks.rados.rados.0.trial169.stdout:22: write initial oid 22 2026-02-25T08:41:34.679 INFO:tasks.rados.rados.0.trial169.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2026-02-25T08:41:34.684 INFO:tasks.rados.rados.0.trial169.stdout:22: writing trial16941936-22 from 752947 to 1365882 tid 1 2026-02-25T08:41:34.689 INFO:tasks.rados.rados.0.trial169.stdout:22: writing trial16941936-22 from 1962710 to 2352947 tid 2 2026-02-25T08:41:34.689 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.689 INFO:tasks.rados.rados.0.trial169.stdout:3: finishing write tid 3 to trial16941936-3 2026-02-25T08:41:34.689 INFO:tasks.rados.rados.0.trial169.stdout:3: finishing write tid 4 to trial16941936-3 2026-02-25T08:41:34.689 INFO:tasks.rados.rados.0.trial169.stdout:3: oid 3 updating version 0 to 1 2026-02-25T08:41:34.689 INFO:tasks.rados.rados.0.trial169.stdout:3: oid 3 updating version 1 to 3 2026-02-25T08:41:34.689 INFO:tasks.rados.rados.0.trial169.stdout:3: oid 3 version 3 is already newer than 3 2026-02-25T08:41:34.689 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2026-02-25T08:41:34.690 INFO:tasks.rados.rados.0.trial169.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2026-02-25T08:41:34.690 INFO:tasks.rados.rados.0.trial169.stdout:14: finishing write tid 1 to trial16941936-14 2026-02-25T08:41:34.690 INFO:tasks.rados.rados.0.trial169.stdout:11: finishing write tid 2 to trial16941936-11 2026-02-25T08:41:34.690 INFO:tasks.rados.rados.0.trial169.stdout:5: finishing write tid 1 to trial16941936-5 2026-02-25T08:41:34.690 INFO:tasks.rados.rados.0.trial169.stdout:11: finishing write tid 3 to trial16941936-11 2026-02-25T08:41:34.690 INFO:tasks.rados.rados.0.trial169.stdout:13: finishing write tid 2 to trial16941936-13 2026-02-25T08:41:34.690 INFO:tasks.rados.rados.0.trial169.stdout:11: finishing write tid 4 to trial16941936-11 2026-02-25T08:41:34.690 INFO:tasks.rados.rados.0.trial169.stdout:11: oid 11 updating version 0 to 3 2026-02-25T08:41:34.690 INFO:tasks.rados.rados.0.trial169.stdout:11: oid 11 version 3 is already newer than 1 2026-02-25T08:41:34.690 INFO:tasks.rados.rados.0.trial169.stdout:11: oid 11 version 3 is already newer than 3 2026-02-25T08:41:34.690 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2026-02-25T08:41:34.690 INFO:tasks.rados.rados.0.trial169.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2026-02-25T08:41:34.690 INFO:tasks.rados.rados.0.trial169.stdout:13: finishing write tid 3 to trial16941936-13 2026-02-25T08:41:34.691 INFO:tasks.rados.rados.0.trial169.stdout:12: finishing write tid 2 to trial16941936-12 2026-02-25T08:41:34.691 INFO:tasks.rados.rados.0.trial169.stdout:3: done (15 left) 2026-02-25T08:41:34.691 INFO:tasks.rados.rados.0.trial169.stdout:11: done (14 left) 2026-02-25T08:41:34.691 INFO:tasks.rados.rados.0.trial169.stdout:23: write initial oid 23 2026-02-25T08:41:34.691 INFO:tasks.rados.rados.0.trial169.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2026-02-25T08:41:34.694 INFO:tasks.rados.rados.0.trial169.stdout:23: writing trial16941936-23 from 769754 to 1457938 tid 1 2026-02-25T08:41:34.697 INFO:tasks.rados.rados.0.trial169.stdout:23: writing trial16941936-23 from 2021192 to 2369754 tid 2 2026-02-25T08:41:34.698 INFO:tasks.rados.rados.0.trial169.stdout:24: write initial oid 24 2026-02-25T08:41:34.698 INFO:tasks.rados.rados.0.trial169.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2026-02-25T08:41:34.702 INFO:tasks.rados.rados.0.trial169.stdout:24: writing trial16941936-24 from 786561 to 1266347 tid 1 2026-02-25T08:41:34.707 INFO:tasks.rados.rados.0.trial169.stdout:24: writing trial16941936-24 from 1796027 to 2263691 tid 2 2026-02-25T08:41:34.707 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.708 INFO:tasks.rados.rados.0.trial169.stdout:7: finishing write tid 2 to trial16941936-7 2026-02-25T08:41:34.708 INFO:tasks.rados.rados.0.trial169.stdout:12: finishing write tid 3 to trial16941936-12 2026-02-25T08:41:34.708 INFO:tasks.rados.rados.0.trial169.stdout:14: finishing write tid 2 to trial16941936-14 2026-02-25T08:41:34.708 INFO:tasks.rados.rados.0.trial169.stdout:17: finishing write tid 1 to trial16941936-17 2026-02-25T08:41:34.708 INFO:tasks.rados.rados.0.trial169.stdout:13: finishing write tid 4 to trial16941936-13 2026-02-25T08:41:34.708 INFO:tasks.rados.rados.0.trial169.stdout:13: oid 13 updating version 0 to 4 2026-02-25T08:41:34.708 INFO:tasks.rados.rados.0.trial169.stdout:13: oid 13 updating version 4 to 6 2026-02-25T08:41:34.708 INFO:tasks.rados.rados.0.trial169.stdout:13: oid 13 version 6 is already newer than 6 2026-02-25T08:41:34.708 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2026-02-25T08:41:34.708 INFO:tasks.rados.rados.0.trial169.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2026-02-25T08:41:34.708 INFO:tasks.rados.rados.0.trial169.stdout:14: finishing write tid 3 to trial16941936-14 2026-02-25T08:41:34.708 INFO:tasks.rados.rados.0.trial169.stdout:12: finishing write tid 4 to trial16941936-12 2026-02-25T08:41:34.709 INFO:tasks.rados.rados.0.trial169.stdout:12: oid 12 updating version 0 to 1 2026-02-25T08:41:34.709 INFO:tasks.rados.rados.0.trial169.stdout:12: oid 12 updating version 1 to 3 2026-02-25T08:41:34.709 INFO:tasks.rados.rados.0.trial169.stdout:12: oid 12 version 3 is already newer than 3 2026-02-25T08:41:34.709 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2026-02-25T08:41:34.709 INFO:tasks.rados.rados.0.trial169.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2026-02-25T08:41:34.709 INFO:tasks.rados.rados.0.trial169.stdout:14: finishing write tid 4 to trial16941936-14 2026-02-25T08:41:34.709 INFO:tasks.rados.rados.0.trial169.stdout:14: oid 14 updating version 0 to 3 2026-02-25T08:41:34.709 INFO:tasks.rados.rados.0.trial169.stdout:14: oid 14 version 3 is already newer than 3 2026-02-25T08:41:34.709 INFO:tasks.rados.rados.0.trial169.stdout:14: oid 14 version 3 is already newer than 1 2026-02-25T08:41:34.709 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2026-02-25T08:41:34.709 INFO:tasks.rados.rados.0.trial169.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2026-02-25T08:41:34.709 INFO:tasks.rados.rados.0.trial169.stdout:7: finishing write tid 3 to trial16941936-7 2026-02-25T08:41:34.710 INFO:tasks.rados.rados.0.trial169.stdout:12: done (15 left) 2026-02-25T08:41:34.710 INFO:tasks.rados.rados.0.trial169.stdout:13: done (14 left) 2026-02-25T08:41:34.710 INFO:tasks.rados.rados.0.trial169.stdout:14: done (13 left) 2026-02-25T08:41:34.710 INFO:tasks.rados.rados.0.trial169.stdout:25: write initial oid 25 2026-02-25T08:41:34.710 INFO:tasks.rados.rados.0.trial169.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2026-02-25T08:41:34.710 INFO:tasks.rados.rados.0.trial169.stdout:25: writing trial16941936-25 from 403368 to 958403 tid 1 2026-02-25T08:41:34.715 INFO:tasks.rados.rados.0.trial169.stdout:25: writing trial16941936-25 from 1454509 to 2182184 tid 2 2026-02-25T08:41:34.715 INFO:tasks.rados.rados.0.trial169.stdout:26: write initial oid 26 2026-02-25T08:41:34.715 INFO:tasks.rados.rados.0.trial169.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2026-02-25T08:41:34.719 INFO:tasks.rados.rados.0.trial169.stdout:26: writing trial16941936-26 from 420175 to 1050459 tid 1 2026-02-25T08:41:34.725 INFO:tasks.rados.rados.0.trial169.stdout:26: writing trial16941936-26 from 1796638 to 2267971 tid 2 2026-02-25T08:41:34.725 INFO:tasks.rados.rados.0.trial169.stdout:27: write initial oid 27 2026-02-25T08:41:34.725 INFO:tasks.rados.rados.0.trial169.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2026-02-25T08:41:34.729 INFO:tasks.rados.rados.0.trial169.stdout:27: writing trial16941936-27 from 436982 to 1142515 tid 1 2026-02-25T08:41:34.735 INFO:tasks.rados.rados.0.trial169.stdout:27: writing trial16941936-27 from 1855120 to 2436982 tid 2 2026-02-25T08:41:34.736 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.736 INFO:tasks.rados.rados.0.trial169.stdout:10: finishing write tid 2 to trial16941936-10 2026-02-25T08:41:34.736 INFO:tasks.rados.rados.0.trial169.stdout:7: finishing write tid 4 to trial16941936-7 2026-02-25T08:41:34.736 INFO:tasks.rados.rados.0.trial169.stdout:7: oid 7 updating version 0 to 4 2026-02-25T08:41:34.736 INFO:tasks.rados.rados.0.trial169.stdout:7: oid 7 updating version 4 to 6 2026-02-25T08:41:34.736 INFO:tasks.rados.rados.0.trial169.stdout:7: oid 7 version 6 is already newer than 6 2026-02-25T08:41:34.736 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 7 v 6 (ObjNum 6 snap 0 seq_num 6) dirty exists 2026-02-25T08:41:34.736 INFO:tasks.rados.rados.0.trial169.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2026-02-25T08:41:34.736 INFO:tasks.rados.rados.0.trial169.stdout:19: finishing write tid 1 to trial16941936-19 2026-02-25T08:41:34.736 INFO:tasks.rados.rados.0.trial169.stdout:10: finishing write tid 3 to trial16941936-10 2026-02-25T08:41:34.736 INFO:tasks.rados.rados.0.trial169.stdout:10: finishing write tid 4 to trial16941936-10 2026-02-25T08:41:34.737 INFO:tasks.rados.rados.0.trial169.stdout:10: oid 10 updating version 0 to 1 2026-02-25T08:41:34.737 INFO:tasks.rados.rados.0.trial169.stdout:10: oid 10 updating version 1 to 3 2026-02-25T08:41:34.737 INFO:tasks.rados.rados.0.trial169.stdout:10: oid 10 version 3 is already newer than 3 2026-02-25T08:41:34.737 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2026-02-25T08:41:34.737 INFO:tasks.rados.rados.0.trial169.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2026-02-25T08:41:34.737 INFO:tasks.rados.rados.0.trial169.stdout:20: finishing write tid 1 to trial16941936-20 2026-02-25T08:41:34.737 INFO:tasks.rados.rados.0.trial169.stdout:17: finishing write tid 2 to trial16941936-17 2026-02-25T08:41:34.737 INFO:tasks.rados.rados.0.trial169.stdout:5: finishing write tid 2 to trial16941936-5 2026-02-25T08:41:34.737 INFO:tasks.rados.rados.0.trial169.stdout:7: done (15 left) 2026-02-25T08:41:34.737 INFO:tasks.rados.rados.0.trial169.stdout:10: done (14 left) 2026-02-25T08:41:34.737 INFO:tasks.rados.rados.0.trial169.stdout:28: write initial oid 28 2026-02-25T08:41:34.737 INFO:tasks.rados.rados.0.trial169.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2026-02-25T08:41:34.741 INFO:tasks.rados.rados.0.trial169.stdout:28: writing trial16941936-28 from 453789 to 1234571 tid 1 2026-02-25T08:41:34.746 INFO:tasks.rados.rados.0.trial169.stdout:28: writing trial16941936-28 from 1913602 to 2388604 tid 2 2026-02-25T08:41:34.746 INFO:tasks.rados.rados.0.trial169.stdout:29: write initial oid 29 2026-02-25T08:41:34.746 INFO:tasks.rados.rados.0.trial169.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2026-02-25T08:41:34.749 INFO:tasks.rados.rados.0.trial169.stdout:29: writing trial16941936-29 from 470596 to 926627 tid 1 2026-02-25T08:41:34.755 INFO:tasks.rados.rados.0.trial169.stdout:29: writing trial16941936-29 from 1572084 to 2190744 tid 2 2026-02-25T08:41:34.755 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.755 INFO:tasks.rados.rados.0.trial169.stdout:19: finishing write tid 2 to trial16941936-19 2026-02-25T08:41:34.756 INFO:tasks.rados.rados.0.trial169.stdout:17: finishing write tid 3 to trial16941936-17 2026-02-25T08:41:34.756 INFO:tasks.rados.rados.0.trial169.stdout:19: finishing write tid 3 to trial16941936-19 2026-02-25T08:41:34.756 INFO:tasks.rados.rados.0.trial169.stdout:5: finishing write tid 3 to trial16941936-5 2026-02-25T08:41:34.756 INFO:tasks.rados.rados.0.trial169.stdout:20: finishing write tid 2 to trial16941936-20 2026-02-25T08:41:34.756 INFO:tasks.rados.rados.0.trial169.stdout:5: finishing write tid 4 to trial16941936-5 2026-02-25T08:41:34.756 INFO:tasks.rados.rados.0.trial169.stdout:5: oid 5 updating version 0 to 4 2026-02-25T08:41:34.756 INFO:tasks.rados.rados.0.trial169.stdout:5: oid 5 updating version 4 to 6 2026-02-25T08:41:34.756 INFO:tasks.rados.rados.0.trial169.stdout:5: oid 5 version 6 is already newer than 6 2026-02-25T08:41:34.756 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 5 v 6 (ObjNum 4 snap 0 seq_num 4) dirty exists 2026-02-25T08:41:34.756 INFO:tasks.rados.rados.0.trial169.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:41:34.756 INFO:tasks.rados.rados.0.trial169.stdout:19: finishing write tid 4 to trial16941936-19 2026-02-25T08:41:34.756 INFO:tasks.rados.rados.0.trial169.stdout:19: oid 19 updating version 0 to 6 2026-02-25T08:41:34.756 INFO:tasks.rados.rados.0.trial169.stdout:19: oid 19 version 6 is already newer than 6 2026-02-25T08:41:34.757 INFO:tasks.rados.rados.0.trial169.stdout:19: oid 19 version 6 is already newer than 4 2026-02-25T08:41:34.757 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2026-02-25T08:41:34.757 INFO:tasks.rados.rados.0.trial169.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2026-02-25T08:41:34.757 INFO:tasks.rados.rados.0.trial169.stdout:17: finishing write tid 4 to trial16941936-17 2026-02-25T08:41:34.757 INFO:tasks.rados.rados.0.trial169.stdout:17: oid 17 updating version 0 to 6 2026-02-25T08:41:34.757 INFO:tasks.rados.rados.0.trial169.stdout:17: oid 17 version 6 is already newer than 6 2026-02-25T08:41:34.757 INFO:tasks.rados.rados.0.trial169.stdout:17: oid 17 version 6 is already newer than 4 2026-02-25T08:41:34.757 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2026-02-25T08:41:34.757 INFO:tasks.rados.rados.0.trial169.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2026-02-25T08:41:34.757 INFO:tasks.rados.rados.0.trial169.stdout:20: finishing write tid 3 to trial16941936-20 2026-02-25T08:41:34.757 INFO:tasks.rados.rados.0.trial169.stdout:5: done (15 left) 2026-02-25T08:41:34.757 INFO:tasks.rados.rados.0.trial169.stdout:17: done (14 left) 2026-02-25T08:41:34.757 INFO:tasks.rados.rados.0.trial169.stdout:19: done (13 left) 2026-02-25T08:41:34.758 INFO:tasks.rados.rados.0.trial169.stdout:30: write initial oid 30 2026-02-25T08:41:34.758 INFO:tasks.rados.rados.0.trial169.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2026-02-25T08:41:34.758 INFO:tasks.rados.rados.0.trial169.stdout:30: writing trial16941936-30 from 487403 to 1018683 tid 1 2026-02-25T08:41:34.762 INFO:tasks.rados.rados.0.trial169.stdout:30: writing trial16941936-30 from 1514213 to 1992884 tid 2 2026-02-25T08:41:34.762 INFO:tasks.rados.rados.0.trial169.stdout:31: write initial oid 31 2026-02-25T08:41:34.763 INFO:tasks.rados.rados.0.trial169.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2026-02-25T08:41:34.766 INFO:tasks.rados.rados.0.trial169.stdout:31: writing trial16941936-31 from 504210 to 1110739 tid 1 2026-02-25T08:41:34.771 INFO:tasks.rados.rados.0.trial169.stdout:31: writing trial16941936-31 from 1572695 to 2195024 tid 2 2026-02-25T08:41:34.771 INFO:tasks.rados.rados.0.trial169.stdout:32: write initial oid 32 2026-02-25T08:41:34.771 INFO:tasks.rados.rados.0.trial169.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2026-02-25T08:41:34.776 INFO:tasks.rados.rados.0.trial169.stdout:32: writing trial16941936-32 from 521017 to 1319148 tid 1 2026-02-25T08:41:34.780 INFO:tasks.rados.rados.0.trial169.stdout:32: writing trial16941936-32 from 1747530 to 2229870 tid 2 2026-02-25T08:41:34.780 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.781 INFO:tasks.rados.rados.0.trial169.stdout:20: finishing write tid 4 to trial16941936-20 2026-02-25T08:41:34.781 INFO:tasks.rados.rados.0.trial169.stdout:20: oid 20 updating version 0 to 3 2026-02-25T08:41:34.781 INFO:tasks.rados.rados.0.trial169.stdout:20: oid 20 version 3 is already newer than 1 2026-02-25T08:41:34.781 INFO:tasks.rados.rados.0.trial169.stdout:20: oid 20 version 3 is already newer than 3 2026-02-25T08:41:34.781 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2026-02-25T08:41:34.781 INFO:tasks.rados.rados.0.trial169.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2026-02-25T08:41:34.781 INFO:tasks.rados.rados.0.trial169.stdout:21: finishing write tid 1 to trial16941936-21 2026-02-25T08:41:34.781 INFO:tasks.rados.rados.0.trial169.stdout:23: finishing write tid 1 to trial16941936-23 2026-02-25T08:41:34.781 INFO:tasks.rados.rados.0.trial169.stdout:15: finishing write tid 1 to trial16941936-15 2026-02-25T08:41:34.781 INFO:tasks.rados.rados.0.trial169.stdout:18: finishing write tid 1 to trial16941936-18 2026-02-25T08:41:34.781 INFO:tasks.rados.rados.0.trial169.stdout:20: done (15 left) 2026-02-25T08:41:34.781 INFO:tasks.rados.rados.0.trial169.stdout:33: write initial oid 33 2026-02-25T08:41:34.782 INFO:tasks.rados.rados.0.trial169.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2026-02-25T08:41:34.783 INFO:tasks.rados.rados.0.trial169.stdout:33: writing trial16941936-33 from 537824 to 1011204 tid 1 2026-02-25T08:41:34.790 INFO:tasks.rados.rados.0.trial169.stdout:33: writing trial16941936-33 from 1806012 to 2432010 tid 2 2026-02-25T08:41:34.790 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.790 INFO:tasks.rados.rados.0.trial169.stdout:23: finishing write tid 2 to trial16941936-23 2026-02-25T08:41:34.790 INFO:tasks.rados.rados.0.trial169.stdout:15: finishing write tid 2 to trial16941936-15 2026-02-25T08:41:34.790 INFO:tasks.rados.rados.0.trial169.stdout:24: finishing write tid 1 to trial16941936-24 2026-02-25T08:41:34.790 INFO:tasks.rados.rados.0.trial169.stdout:23: finishing write tid 3 to trial16941936-23 2026-02-25T08:41:34.790 INFO:tasks.rados.rados.0.trial169.stdout:23: finishing write tid 4 to trial16941936-23 2026-02-25T08:41:34.790 INFO:tasks.rados.rados.0.trial169.stdout:23: oid 23 updating version 0 to 9 2026-02-25T08:41:34.791 INFO:tasks.rados.rados.0.trial169.stdout:23: oid 23 version 9 is already newer than 9 2026-02-25T08:41:34.791 INFO:tasks.rados.rados.0.trial169.stdout:23: oid 23 version 9 is already newer than 7 2026-02-25T08:41:34.791 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2026-02-25T08:41:34.791 INFO:tasks.rados.rados.0.trial169.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2026-02-25T08:41:34.791 INFO:tasks.rados.rados.0.trial169.stdout:15: finishing write tid 3 to trial16941936-15 2026-02-25T08:41:34.791 INFO:tasks.rados.rados.0.trial169.stdout:23: done (15 left) 2026-02-25T08:41:34.791 INFO:tasks.rados.rados.0.trial169.stdout:34: write initial oid 34 2026-02-25T08:41:34.791 INFO:tasks.rados.rados.0.trial169.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2026-02-25T08:41:34.794 INFO:tasks.rados.rados.0.trial169.stdout:34: writing trial16941936-34 from 554631 to 1103260 tid 1 2026-02-25T08:41:34.799 INFO:tasks.rados.rados.0.trial169.stdout:34: writing trial16941936-34 from 1748141 to 2234150 tid 2 2026-02-25T08:41:34.800 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.800 INFO:tasks.rados.rados.0.trial169.stdout:25: finishing write tid 1 to trial16941936-25 2026-02-25T08:41:34.800 INFO:tasks.rados.rados.0.trial169.stdout:21: finishing write tid 2 to trial16941936-21 2026-02-25T08:41:34.800 INFO:tasks.rados.rados.0.trial169.stdout:15: finishing write tid 4 to trial16941936-15 2026-02-25T08:41:34.800 INFO:tasks.rados.rados.0.trial169.stdout:15: oid 15 updating version 0 to 1 2026-02-25T08:41:34.800 INFO:tasks.rados.rados.0.trial169.stdout:15: oid 15 updating version 1 to 3 2026-02-25T08:41:34.800 INFO:tasks.rados.rados.0.trial169.stdout:15: oid 15 version 3 is already newer than 3 2026-02-25T08:41:34.800 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2026-02-25T08:41:34.800 INFO:tasks.rados.rados.0.trial169.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2026-02-25T08:41:34.800 INFO:tasks.rados.rados.0.trial169.stdout:25: finishing write tid 2 to trial16941936-25 2026-02-25T08:41:34.800 INFO:tasks.rados.rados.0.trial169.stdout:26: finishing write tid 1 to trial16941936-26 2026-02-25T08:41:34.801 INFO:tasks.rados.rados.0.trial169.stdout:15: done (15 left) 2026-02-25T08:41:34.801 INFO:tasks.rados.rados.0.trial169.stdout:35: write initial oid 35 2026-02-25T08:41:34.801 INFO:tasks.rados.rados.0.trial169.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2026-02-25T08:41:34.803 INFO:tasks.rados.rados.0.trial169.stdout:35: writing trial16941936-35 from 571438 to 1195316 tid 1 2026-02-25T08:41:34.809 INFO:tasks.rados.rados.0.trial169.stdout:35: writing trial16941936-35 from 1806623 to 2436290 tid 2 2026-02-25T08:41:34.810 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.810 INFO:tasks.rados.rados.0.trial169.stdout:21: finishing write tid 3 to trial16941936-21 2026-02-25T08:41:34.810 INFO:tasks.rados.rados.0.trial169.stdout:25: finishing write tid 3 to trial16941936-25 2026-02-25T08:41:34.810 INFO:tasks.rados.rados.0.trial169.stdout:25: finishing write tid 4 to trial16941936-25 2026-02-25T08:41:34.810 INFO:tasks.rados.rados.0.trial169.stdout:25: oid 25 updating version 0 to 6 2026-02-25T08:41:34.810 INFO:tasks.rados.rados.0.trial169.stdout:25: oid 25 version 6 is already newer than 4 2026-02-25T08:41:34.810 INFO:tasks.rados.rados.0.trial169.stdout:25: oid 25 version 6 is already newer than 6 2026-02-25T08:41:34.810 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 25 v 6 (ObjNum 24 snap 0 seq_num 24) dirty exists 2026-02-25T08:41:34.810 INFO:tasks.rados.rados.0.trial169.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2026-02-25T08:41:34.810 INFO:tasks.rados.rados.0.trial169.stdout:21: finishing write tid 4 to trial16941936-21 2026-02-25T08:41:34.810 INFO:tasks.rados.rados.0.trial169.stdout:21: oid 21 updating version 0 to 9 2026-02-25T08:41:34.811 INFO:tasks.rados.rados.0.trial169.stdout:21: oid 21 version 9 is already newer than 9 2026-02-25T08:41:34.811 INFO:tasks.rados.rados.0.trial169.stdout:21: oid 21 version 9 is already newer than 7 2026-02-25T08:41:34.811 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2026-02-25T08:41:34.811 INFO:tasks.rados.rados.0.trial169.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2026-02-25T08:41:34.811 INFO:tasks.rados.rados.0.trial169.stdout:26: finishing write tid 2 to trial16941936-26 2026-02-25T08:41:34.811 INFO:tasks.rados.rados.0.trial169.stdout:27: finishing write tid 1 to trial16941936-27 2026-02-25T08:41:34.811 INFO:tasks.rados.rados.0.trial169.stdout:16: finishing write tid 1 to trial16941936-16 2026-02-25T08:41:34.811 INFO:tasks.rados.rados.0.trial169.stdout:18: finishing write tid 2 to trial16941936-18 2026-02-25T08:41:34.811 INFO:tasks.rados.rados.0.trial169.stdout:29: finishing write tid 1 to trial16941936-29 2026-02-25T08:41:34.811 INFO:tasks.rados.rados.0.trial169.stdout:26: finishing write tid 3 to trial16941936-26 2026-02-25T08:41:34.811 INFO:tasks.rados.rados.0.trial169.stdout:24: finishing write tid 2 to trial16941936-24 2026-02-25T08:41:34.811 INFO:tasks.rados.rados.0.trial169.stdout:21: done (15 left) 2026-02-25T08:41:34.811 INFO:tasks.rados.rados.0.trial169.stdout:25: done (14 left) 2026-02-25T08:41:34.812 INFO:tasks.rados.rados.0.trial169.stdout:36: write initial oid 36 2026-02-25T08:41:34.812 INFO:tasks.rados.rados.0.trial169.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2026-02-25T08:41:34.815 INFO:tasks.rados.rados.0.trial169.stdout:36: writing trial16941936-36 from 588245 to 1287372 tid 1 2026-02-25T08:41:34.819 INFO:tasks.rados.rados.0.trial169.stdout:36: writing trial16941936-36 from 1865105 to 2354783 tid 2 2026-02-25T08:41:34.819 INFO:tasks.rados.rados.0.trial169.stdout:37: write initial oid 37 2026-02-25T08:41:34.820 INFO:tasks.rados.rados.0.trial169.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2026-02-25T08:41:34.825 INFO:tasks.rados.rados.0.trial169.stdout:37: writing trial16941936-37 from 605052 to 1379428 tid 1 2026-02-25T08:41:34.830 INFO:tasks.rados.rados.0.trial169.stdout:37: writing trial16941936-37 from 1923587 to 2556923 tid 2 2026-02-25T08:41:34.830 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.830 INFO:tasks.rados.rados.0.trial169.stdout:26: finishing write tid 4 to trial16941936-26 2026-02-25T08:41:34.830 INFO:tasks.rados.rados.0.trial169.stdout:26: oid 26 updating version 0 to 9 2026-02-25T08:41:34.830 INFO:tasks.rados.rados.0.trial169.stdout:26: oid 26 version 9 is already newer than 9 2026-02-25T08:41:34.830 INFO:tasks.rados.rados.0.trial169.stdout:26: oid 26 version 9 is already newer than 7 2026-02-25T08:41:34.830 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 26 v 9 (ObjNum 25 snap 0 seq_num 25) dirty exists 2026-02-25T08:41:34.830 INFO:tasks.rados.rados.0.trial169.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2026-02-25T08:41:34.830 INFO:tasks.rados.rados.0.trial169.stdout:30: finishing write tid 1 to trial16941936-30 2026-02-25T08:41:34.831 INFO:tasks.rados.rados.0.trial169.stdout:26: done (15 left) 2026-02-25T08:41:34.831 INFO:tasks.rados.rados.0.trial169.stdout:38: write initial oid 38 2026-02-25T08:41:34.831 INFO:tasks.rados.rados.0.trial169.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2026-02-25T08:41:34.833 INFO:tasks.rados.rados.0.trial169.stdout:38: writing trial16941936-38 from 621859 to 1071484 tid 1 2026-02-25T08:41:34.840 INFO:tasks.rados.rados.0.trial169.stdout:38: writing trial16941936-38 from 1865716 to 2621859 tid 2 2026-02-25T08:41:34.840 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.841 INFO:tasks.rados.rados.0.trial169.stdout:16: finishing write tid 2 to trial16941936-16 2026-02-25T08:41:34.841 INFO:tasks.rados.rados.0.trial169.stdout:27: finishing write tid 2 to trial16941936-27 2026-02-25T08:41:34.841 INFO:tasks.rados.rados.0.trial169.stdout:18: finishing write tid 3 to trial16941936-18 2026-02-25T08:41:34.841 INFO:tasks.rados.rados.0.trial169.stdout:29: finishing write tid 2 to trial16941936-29 2026-02-25T08:41:34.841 INFO:tasks.rados.rados.0.trial169.stdout:30: finishing write tid 2 to trial16941936-30 2026-02-25T08:41:34.841 INFO:tasks.rados.rados.0.trial169.stdout:27: finishing write tid 3 to trial16941936-27 2026-02-25T08:41:34.841 INFO:tasks.rados.rados.0.trial169.stdout:31: finishing write tid 1 to trial16941936-31 2026-02-25T08:41:34.841 INFO:tasks.rados.rados.0.trial169.stdout:18: finishing write tid 4 to trial16941936-18 2026-02-25T08:41:34.841 INFO:tasks.rados.rados.0.trial169.stdout:18: oid 18 updating version 0 to 9 2026-02-25T08:41:34.841 INFO:tasks.rados.rados.0.trial169.stdout:18: oid 18 version 9 is already newer than 7 2026-02-25T08:41:34.841 INFO:tasks.rados.rados.0.trial169.stdout:18: oid 18 version 9 is already newer than 9 2026-02-25T08:41:34.841 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 18 v 9 (ObjNum 17 snap 0 seq_num 17) dirty exists 2026-02-25T08:41:34.842 INFO:tasks.rados.rados.0.trial169.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2026-02-25T08:41:34.842 INFO:tasks.rados.rados.0.trial169.stdout:27: finishing write tid 4 to trial16941936-27 2026-02-25T08:41:34.842 INFO:tasks.rados.rados.0.trial169.stdout:27: oid 27 updating version 0 to 6 2026-02-25T08:41:34.842 INFO:tasks.rados.rados.0.trial169.stdout:27: oid 27 version 6 is already newer than 4 2026-02-25T08:41:34.842 INFO:tasks.rados.rados.0.trial169.stdout:27: oid 27 version 6 is already newer than 6 2026-02-25T08:41:34.842 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2026-02-25T08:41:34.842 INFO:tasks.rados.rados.0.trial169.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2026-02-25T08:41:34.842 INFO:tasks.rados.rados.0.trial169.stdout:32: finishing write tid 1 to trial16941936-32 2026-02-25T08:41:34.842 INFO:tasks.rados.rados.0.trial169.stdout:18: done (15 left) 2026-02-25T08:41:34.842 INFO:tasks.rados.rados.0.trial169.stdout:27: done (14 left) 2026-02-25T08:41:34.842 INFO:tasks.rados.rados.0.trial169.stdout:39: write initial oid 39 2026-02-25T08:41:34.842 INFO:tasks.rados.rados.0.trial169.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2026-02-25T08:41:34.844 INFO:tasks.rados.rados.0.trial169.stdout:39: writing trial16941936-39 from 638666 to 1163540 tid 1 2026-02-25T08:41:34.850 INFO:tasks.rados.rados.0.trial169.stdout:39: writing trial16941936-39 from 1924198 to 2561203 tid 2 2026-02-25T08:41:34.850 INFO:tasks.rados.rados.0.trial169.stdout:40: write initial oid 40 2026-02-25T08:41:34.850 INFO:tasks.rados.rados.0.trial169.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2026-02-25T08:41:34.855 INFO:tasks.rados.rados.0.trial169.stdout:40: writing trial16941936-40 from 655473 to 1371949 tid 1 2026-02-25T08:41:34.861 INFO:tasks.rados.rados.0.trial169.stdout:40: writing trial16941936-40 from 2099033 to 2655473 tid 2 2026-02-25T08:41:34.861 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.861 INFO:tasks.rados.rados.0.trial169.stdout:30: finishing write tid 3 to trial16941936-30 2026-02-25T08:41:34.861 INFO:tasks.rados.rados.0.trial169.stdout:30: finishing write tid 4 to trial16941936-30 2026-02-25T08:41:34.861 INFO:tasks.rados.rados.0.trial169.stdout:30: oid 30 updating version 0 to 6 2026-02-25T08:41:34.861 INFO:tasks.rados.rados.0.trial169.stdout:30: oid 30 version 6 is already newer than 6 2026-02-25T08:41:34.861 INFO:tasks.rados.rados.0.trial169.stdout:30: oid 30 version 6 is already newer than 4 2026-02-25T08:41:34.861 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 30 v 6 (ObjNum 29 snap 0 seq_num 29) dirty exists 2026-02-25T08:41:34.862 INFO:tasks.rados.rados.0.trial169.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2026-02-25T08:41:34.862 INFO:tasks.rados.rados.0.trial169.stdout:29: finishing write tid 3 to trial16941936-29 2026-02-25T08:41:34.862 INFO:tasks.rados.rados.0.trial169.stdout:29: finishing write tid 4 to trial16941936-29 2026-02-25T08:41:34.862 INFO:tasks.rados.rados.0.trial169.stdout:29: oid 29 updating version 0 to 9 2026-02-25T08:41:34.862 INFO:tasks.rados.rados.0.trial169.stdout:29: oid 29 version 9 is already newer than 9 2026-02-25T08:41:34.862 INFO:tasks.rados.rados.0.trial169.stdout:29: oid 29 version 9 is already newer than 7 2026-02-25T08:41:34.862 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 29 v 9 (ObjNum 28 snap 0 seq_num 28) dirty exists 2026-02-25T08:41:34.862 INFO:tasks.rados.rados.0.trial169.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2026-02-25T08:41:34.862 INFO:tasks.rados.rados.0.trial169.stdout:34: finishing write tid 1 to trial16941936-34 2026-02-25T08:41:34.862 INFO:tasks.rados.rados.0.trial169.stdout:28: finishing write tid 1 to trial16941936-28 2026-02-25T08:41:34.862 INFO:tasks.rados.rados.0.trial169.stdout:31: finishing write tid 2 to trial16941936-31 2026-02-25T08:41:34.862 INFO:tasks.rados.rados.0.trial169.stdout:16: finishing write tid 3 to trial16941936-16 2026-02-25T08:41:34.863 INFO:tasks.rados.rados.0.trial169.stdout:29: done (15 left) 2026-02-25T08:41:34.863 INFO:tasks.rados.rados.0.trial169.stdout:30: done (14 left) 2026-02-25T08:41:34.863 INFO:tasks.rados.rados.0.trial169.stdout:41: write initial oid 41 2026-02-25T08:41:34.863 INFO:tasks.rados.rados.0.trial169.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2026-02-25T08:41:34.865 INFO:tasks.rados.rados.0.trial169.stdout:41: writing trial16941936-41 from 672280 to 1464005 tid 1 2026-02-25T08:41:34.870 INFO:tasks.rados.rados.0.trial169.stdout:41: writing trial16941936-41 from 2157515 to 2672280 tid 2 2026-02-25T08:41:34.870 INFO:tasks.rados.rados.0.trial169.stdout:42: write initial oid 42 2026-02-25T08:41:34.870 INFO:tasks.rados.rados.0.trial169.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2026-02-25T08:41:34.874 INFO:tasks.rados.rados.0.trial169.stdout:42: writing trial16941936-42 from 689087 to 1156061 tid 1 2026-02-25T08:41:34.880 INFO:tasks.rados.rados.0.trial169.stdout:42: writing trial16941936-42 from 1699644 to 2483976 tid 2 2026-02-25T08:41:34.881 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.881 INFO:tasks.rados.rados.0.trial169.stdout:16: finishing write tid 4 to trial16941936-16 2026-02-25T08:41:34.881 INFO:tasks.rados.rados.0.trial169.stdout:16: oid 16 updating version 0 to 6 2026-02-25T08:41:34.881 INFO:tasks.rados.rados.0.trial169.stdout:16: oid 16 version 6 is already newer than 4 2026-02-25T08:41:34.881 INFO:tasks.rados.rados.0.trial169.stdout:16: oid 16 version 6 is already newer than 6 2026-02-25T08:41:34.881 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2026-02-25T08:41:34.881 INFO:tasks.rados.rados.0.trial169.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2026-02-25T08:41:34.881 INFO:tasks.rados.rados.0.trial169.stdout:31: finishing write tid 3 to trial16941936-31 2026-02-25T08:41:34.881 INFO:tasks.rados.rados.0.trial169.stdout:31: finishing write tid 4 to trial16941936-31 2026-02-25T08:41:34.881 INFO:tasks.rados.rados.0.trial169.stdout:31: oid 31 updating version 0 to 12 2026-02-25T08:41:34.881 INFO:tasks.rados.rados.0.trial169.stdout:31: oid 31 version 12 is already newer than 10 2026-02-25T08:41:34.882 INFO:tasks.rados.rados.0.trial169.stdout:31: oid 31 version 12 is already newer than 12 2026-02-25T08:41:34.882 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 31 v 12 (ObjNum 30 snap 0 seq_num 30) dirty exists 2026-02-25T08:41:34.882 INFO:tasks.rados.rados.0.trial169.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2026-02-25T08:41:34.882 INFO:tasks.rados.rados.0.trial169.stdout:36: finishing write tid 1 to trial16941936-36 2026-02-25T08:41:34.882 INFO:tasks.rados.rados.0.trial169.stdout:16: done (15 left) 2026-02-25T08:41:34.882 INFO:tasks.rados.rados.0.trial169.stdout:31: done (14 left) 2026-02-25T08:41:34.882 INFO:tasks.rados.rados.0.trial169.stdout:43: write initial oid 43 2026-02-25T08:41:34.882 INFO:tasks.rados.rados.0.trial169.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2026-02-25T08:41:34.884 INFO:tasks.rados.rados.0.trial169.stdout:43: writing trial16941936-43 from 705894 to 1248117 tid 1 2026-02-25T08:41:34.889 INFO:tasks.rados.rados.0.trial169.stdout:43: writing trial16941936-43 from 1758126 to 2402469 tid 2 2026-02-25T08:41:34.889 INFO:tasks.rados.rados.0.trial169.stdout:44: write initial oid 44 2026-02-25T08:41:34.890 INFO:tasks.rados.rados.0.trial169.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2026-02-25T08:41:34.894 INFO:tasks.rados.rados.0.trial169.stdout:44: writing trial16941936-44 from 722701 to 1340173 tid 1 2026-02-25T08:41:34.900 INFO:tasks.rados.rados.0.trial169.stdout:44: writing trial16941936-44 from 1816608 to 2604609 tid 2 2026-02-25T08:41:34.900 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.900 INFO:tasks.rados.rados.0.trial169.stdout:34: finishing write tid 2 to trial16941936-34 2026-02-25T08:41:34.900 INFO:tasks.rados.rados.0.trial169.stdout:32: finishing write tid 2 to trial16941936-32 2026-02-25T08:41:34.900 INFO:tasks.rados.rados.0.trial169.stdout:22: finishing write tid 1 to trial16941936-22 2026-02-25T08:41:34.900 INFO:tasks.rados.rados.0.trial169.stdout:37: finishing write tid 1 to trial16941936-37 2026-02-25T08:41:34.900 INFO:tasks.rados.rados.0.trial169.stdout:33: finishing write tid 1 to trial16941936-33 2026-02-25T08:41:34.900 INFO:tasks.rados.rados.0.trial169.stdout:32: finishing write tid 3 to trial16941936-32 2026-02-25T08:41:34.901 INFO:tasks.rados.rados.0.trial169.stdout:34: finishing write tid 3 to trial16941936-34 2026-02-25T08:41:34.901 INFO:tasks.rados.rados.0.trial169.stdout:34: finishing write tid 4 to trial16941936-34 2026-02-25T08:41:34.901 INFO:tasks.rados.rados.0.trial169.stdout:34: oid 34 updating version 0 to 10 2026-02-25T08:41:34.901 INFO:tasks.rados.rados.0.trial169.stdout:34: oid 34 updating version 10 to 12 2026-02-25T08:41:34.901 INFO:tasks.rados.rados.0.trial169.stdout:34: oid 34 version 12 is already newer than 12 2026-02-25T08:41:34.901 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 34 v 12 (ObjNum 33 snap 0 seq_num 33) dirty exists 2026-02-25T08:41:34.901 INFO:tasks.rados.rados.0.trial169.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2026-02-25T08:41:34.901 INFO:tasks.rados.rados.0.trial169.stdout:32: finishing write tid 4 to trial16941936-32 2026-02-25T08:41:34.901 INFO:tasks.rados.rados.0.trial169.stdout:32: oid 32 updating version 0 to 10 2026-02-25T08:41:34.901 INFO:tasks.rados.rados.0.trial169.stdout:32: oid 32 updating version 10 to 12 2026-02-25T08:41:34.901 INFO:tasks.rados.rados.0.trial169.stdout:32: oid 32 version 12 is already newer than 12 2026-02-25T08:41:34.901 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 32 v 12 (ObjNum 31 snap 0 seq_num 31) dirty exists 2026-02-25T08:41:34.902 INFO:tasks.rados.rados.0.trial169.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2026-02-25T08:41:34.902 INFO:tasks.rados.rados.0.trial169.stdout:36: finishing write tid 2 to trial16941936-36 2026-02-25T08:41:34.902 INFO:tasks.rados.rados.0.trial169.stdout:32: done (15 left) 2026-02-25T08:41:34.902 INFO:tasks.rados.rados.0.trial169.stdout:34: done (14 left) 2026-02-25T08:41:34.902 INFO:tasks.rados.rados.0.trial169.stdout:45: write initial oid 45 2026-02-25T08:41:34.902 INFO:tasks.rados.rados.0.trial169.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2026-02-25T08:41:34.905 INFO:tasks.rados.rados.0.trial169.stdout:45: writing trial16941936-45 from 739508 to 1432229 tid 1 2026-02-25T08:41:34.909 INFO:tasks.rados.rados.0.trial169.stdout:45: writing trial16941936-45 from 1875090 to 2523102 tid 2 2026-02-25T08:41:34.909 INFO:tasks.rados.rados.0.trial169.stdout:46: write initial oid 46 2026-02-25T08:41:34.909 INFO:tasks.rados.rados.0.trial169.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2026-02-25T08:41:34.915 INFO:tasks.rados.rados.0.trial169.stdout:46: writing trial16941936-46 from 756315 to 1524285 tid 1 2026-02-25T08:41:34.920 INFO:tasks.rados.rados.0.trial169.stdout:46: writing trial16941936-46 from 1933572 to 2725242 tid 2 2026-02-25T08:41:34.920 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.921 INFO:tasks.rados.rados.0.trial169.stdout:37: finishing write tid 2 to trial16941936-37 2026-02-25T08:41:34.921 INFO:tasks.rados.rados.0.trial169.stdout:22: finishing write tid 2 to trial16941936-22 2026-02-25T08:41:34.921 INFO:tasks.rados.rados.0.trial169.stdout:28: finishing write tid 2 to trial16941936-28 2026-02-25T08:41:34.921 INFO:tasks.rados.rados.0.trial169.stdout:36: finishing write tid 3 to trial16941936-36 2026-02-25T08:41:34.921 INFO:tasks.rados.rados.0.trial169.stdout:36: finishing write tid 4 to trial16941936-36 2026-02-25T08:41:34.921 INFO:tasks.rados.rados.0.trial169.stdout:36: oid 36 updating version 0 to 7 2026-02-25T08:41:34.921 INFO:tasks.rados.rados.0.trial169.stdout:36: oid 36 updating version 7 to 9 2026-02-25T08:41:34.921 INFO:tasks.rados.rados.0.trial169.stdout:36: oid 36 version 9 is already newer than 9 2026-02-25T08:41:34.921 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 36 v 9 (ObjNum 35 snap 0 seq_num 35) dirty exists 2026-02-25T08:41:34.921 INFO:tasks.rados.rados.0.trial169.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:41:34.921 INFO:tasks.rados.rados.0.trial169.stdout:39: finishing write tid 1 to trial16941936-39 2026-02-25T08:41:34.922 INFO:tasks.rados.rados.0.trial169.stdout:36: done (15 left) 2026-02-25T08:41:34.922 INFO:tasks.rados.rados.0.trial169.stdout:47: write initial oid 47 2026-02-25T08:41:34.922 INFO:tasks.rados.rados.0.trial169.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2026-02-25T08:41:34.925 INFO:tasks.rados.rados.0.trial169.stdout:47: writing trial16941936-47 from 773122 to 1332694 tid 1 2026-02-25T08:41:34.930 INFO:tasks.rados.rados.0.trial169.stdout:47: writing trial16941936-47 from 1992054 to 2643735 tid 2 2026-02-25T08:41:34.931 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.931 INFO:tasks.rados.rados.0.trial169.stdout:24: finishing write tid 3 to trial16941936-24 2026-02-25T08:41:34.931 INFO:tasks.rados.rados.0.trial169.stdout:22: finishing write tid 3 to trial16941936-22 2026-02-25T08:41:34.931 INFO:tasks.rados.rados.0.trial169.stdout:24: finishing write tid 4 to trial16941936-24 2026-02-25T08:41:34.931 INFO:tasks.rados.rados.0.trial169.stdout:24: oid 24 updating version 0 to 6 2026-02-25T08:41:34.931 INFO:tasks.rados.rados.0.trial169.stdout:24: oid 24 version 6 is already newer than 6 2026-02-25T08:41:34.931 INFO:tasks.rados.rados.0.trial169.stdout:24: oid 24 version 6 is already newer than 4 2026-02-25T08:41:34.931 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 24 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2026-02-25T08:41:34.931 INFO:tasks.rados.rados.0.trial169.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2026-02-25T08:41:34.931 INFO:tasks.rados.rados.0.trial169.stdout:38: finishing write tid 1 to trial16941936-38 2026-02-25T08:41:34.931 INFO:tasks.rados.rados.0.trial169.stdout:24: done (15 left) 2026-02-25T08:41:34.932 INFO:tasks.rados.rados.0.trial169.stdout:48: write initial oid 48 2026-02-25T08:41:34.932 INFO:tasks.rados.rados.0.trial169.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2026-02-25T08:41:34.935 INFO:tasks.rados.rados.0.trial169.stdout:48: writing trial16941936-48 from 789929 to 1424750 tid 1 2026-02-25T08:41:34.941 INFO:tasks.rados.rados.0.trial169.stdout:48: writing trial16941936-48 from 2050536 to 2789929 tid 2 2026-02-25T08:41:34.941 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.941 INFO:tasks.rados.rados.0.trial169.stdout:22: finishing write tid 4 to trial16941936-22 2026-02-25T08:41:34.941 INFO:tasks.rados.rados.0.trial169.stdout:22: oid 22 updating version 0 to 9 2026-02-25T08:41:34.941 INFO:tasks.rados.rados.0.trial169.stdout:22: oid 22 version 9 is already newer than 7 2026-02-25T08:41:34.942 INFO:tasks.rados.rados.0.trial169.stdout:22: oid 22 version 9 is already newer than 9 2026-02-25T08:41:34.942 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 22 v 9 (ObjNum 21 snap 0 seq_num 21) dirty exists 2026-02-25T08:41:34.942 INFO:tasks.rados.rados.0.trial169.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2026-02-25T08:41:34.942 INFO:tasks.rados.rados.0.trial169.stdout:28: finishing write tid 3 to trial16941936-28 2026-02-25T08:41:34.942 INFO:tasks.rados.rados.0.trial169.stdout:22: done (15 left) 2026-02-25T08:41:34.942 INFO:tasks.rados.rados.0.trial169.stdout:49: write initial oid 49 2026-02-25T08:41:34.942 INFO:tasks.rados.rados.0.trial169.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2026-02-25T08:41:34.945 INFO:tasks.rados.rados.0.trial169.stdout:49: writing trial16941936-49 from 406736 to 1116806 tid 1 2026-02-25T08:41:34.950 INFO:tasks.rados.rados.0.trial169.stdout:49: writing trial16941936-49 from 1709018 to 2364368 tid 2 2026-02-25T08:41:34.950 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.951 INFO:tasks.rados.rados.0.trial169.stdout:33: finishing write tid 2 to trial16941936-33 2026-02-25T08:41:34.951 INFO:tasks.rados.rados.0.trial169.stdout:28: finishing write tid 4 to trial16941936-28 2026-02-25T08:41:34.951 INFO:tasks.rados.rados.0.trial169.stdout:28: oid 28 updating version 0 to 6 2026-02-25T08:41:34.951 INFO:tasks.rados.rados.0.trial169.stdout:28: oid 28 version 6 is already newer than 6 2026-02-25T08:41:34.951 INFO:tasks.rados.rados.0.trial169.stdout:28: oid 28 version 6 is already newer than 4 2026-02-25T08:41:34.951 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2026-02-25T08:41:34.951 INFO:tasks.rados.rados.0.trial169.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2026-02-25T08:41:34.951 INFO:tasks.rados.rados.0.trial169.stdout:33: finishing write tid 3 to trial16941936-33 2026-02-25T08:41:34.951 INFO:tasks.rados.rados.0.trial169.stdout:33: finishing write tid 4 to trial16941936-33 2026-02-25T08:41:34.951 INFO:tasks.rados.rados.0.trial169.stdout:33: oid 33 updating version 0 to 12 2026-02-25T08:41:34.951 INFO:tasks.rados.rados.0.trial169.stdout:33: oid 33 version 12 is already newer than 12 2026-02-25T08:41:34.951 INFO:tasks.rados.rados.0.trial169.stdout:33: oid 33 version 12 is already newer than 10 2026-02-25T08:41:34.952 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 33 v 12 (ObjNum 32 snap 0 seq_num 32) dirty exists 2026-02-25T08:41:34.952 INFO:tasks.rados.rados.0.trial169.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2026-02-25T08:41:34.952 INFO:tasks.rados.rados.0.trial169.stdout:37: finishing write tid 3 to trial16941936-37 2026-02-25T08:41:34.952 INFO:tasks.rados.rados.0.trial169.stdout:28: done (15 left) 2026-02-25T08:41:34.952 INFO:tasks.rados.rados.0.trial169.stdout:33: done (14 left) 2026-02-25T08:41:34.952 INFO:tasks.rados.rados.0.trial169.stdout:50: write initial oid 50 2026-02-25T08:41:34.952 INFO:tasks.rados.rados.0.trial169.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2026-02-25T08:41:34.954 INFO:tasks.rados.rados.0.trial169.stdout:50: writing trial16941936-50 from 423543 to 1208862 tid 1 2026-02-25T08:41:34.960 INFO:tasks.rados.rados.0.trial169.stdout:50: writing trial16941936-50 from 1767500 to 2566508 tid 2 2026-02-25T08:41:34.960 INFO:tasks.rados.rados.0.trial169.stdout:51: write (excl) oid 2 current snap is 0 2026-02-25T08:41:34.960 INFO:tasks.rados.rados.0.trial169.stdout:51: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2026-02-25T08:41:34.963 INFO:tasks.rados.rados.0.trial169.stdout:51: writing trial16941936-2 from 440350 to 900918 tid 1 2026-02-25T08:41:34.967 INFO:tasks.rados.rados.0.trial169.stdout:51: writing trial16941936-2 from 1309629 to 1852295 tid 2 2026-02-25T08:41:34.971 INFO:tasks.rados.rados.0.trial169.stdout:51: writing trial16941936-2 from 2323973 to 2840350 tid 3 2026-02-25T08:41:34.971 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.972 INFO:tasks.rados.rados.0.trial169.stdout:39: finishing write tid 2 to trial16941936-39 2026-02-25T08:41:34.972 INFO:tasks.rados.rados.0.trial169.stdout:37: finishing write tid 4 to trial16941936-37 2026-02-25T08:41:34.972 INFO:tasks.rados.rados.0.trial169.stdout:37: oid 37 updating version 0 to 6 2026-02-25T08:41:34.972 INFO:tasks.rados.rados.0.trial169.stdout:37: oid 37 version 6 is already newer than 6 2026-02-25T08:41:34.972 INFO:tasks.rados.rados.0.trial169.stdout:37: oid 37 version 6 is already newer than 4 2026-02-25T08:41:34.972 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2026-02-25T08:41:34.972 INFO:tasks.rados.rados.0.trial169.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2026-02-25T08:41:34.972 INFO:tasks.rados.rados.0.trial169.stdout:41: finishing write tid 1 to trial16941936-41 2026-02-25T08:41:34.972 INFO:tasks.rados.rados.0.trial169.stdout:37: done (15 left) 2026-02-25T08:41:34.972 INFO:tasks.rados.rados.0.trial169.stdout:52: write (excl) oid 14 current snap is 0 2026-02-25T08:41:34.972 INFO:tasks.rados.rados.0.trial169.stdout:39: finishing write tid 3 to trial16941936-39 2026-02-25T08:41:34.973 INFO:tasks.rados.rados.0.trial169.stdout:43: finishing write tid 1 to trial16941936-43 2026-02-25T08:41:34.973 INFO:tasks.rados.rados.0.trial169.stdout:52: seq_num 51 ranges {457157=535817,1768111=402677} 2026-02-25T08:41:34.974 INFO:tasks.rados.rados.0.trial169.stdout:52: writing trial16941936-14 from 457157 to 992974 tid 1 2026-02-25T08:41:34.982 INFO:tasks.rados.rados.0.trial169.stdout:52: writing trial16941936-14 from 1768111 to 2170788 tid 2 2026-02-25T08:41:34.982 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.982 INFO:tasks.rados.rados.0.trial169.stdout:35: finishing write tid 1 to trial16941936-35 2026-02-25T08:41:34.982 INFO:tasks.rados.rados.0.trial169.stdout:39: finishing write tid 4 to trial16941936-39 2026-02-25T08:41:34.982 INFO:tasks.rados.rados.0.trial169.stdout:39: oid 39 updating version 0 to 10 2026-02-25T08:41:34.983 INFO:tasks.rados.rados.0.trial169.stdout:39: oid 39 updating version 10 to 12 2026-02-25T08:41:34.983 INFO:tasks.rados.rados.0.trial169.stdout:39: oid 39 version 12 is already newer than 12 2026-02-25T08:41:34.983 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 39 v 12 (ObjNum 38 snap 0 seq_num 38) dirty exists 2026-02-25T08:41:34.983 INFO:tasks.rados.rados.0.trial169.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2026-02-25T08:41:34.983 INFO:tasks.rados.rados.0.trial169.stdout:38: finishing write tid 2 to trial16941936-38 2026-02-25T08:41:34.983 INFO:tasks.rados.rados.0.trial169.stdout:41: finishing write tid 2 to trial16941936-41 2026-02-25T08:41:34.983 INFO:tasks.rados.rados.0.trial169.stdout:39: done (15 left) 2026-02-25T08:41:34.983 INFO:tasks.rados.rados.0.trial169.stdout:53: read oid 20 snap -1 2026-02-25T08:41:34.983 INFO:tasks.rados.rados.0.trial169.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2026-02-25T08:41:34.983 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.983 INFO:tasks.rados.rados.0.trial169.stdout:43: finishing write tid 2 to trial16941936-43 2026-02-25T08:41:34.983 INFO:tasks.rados.rados.0.trial169.stdout:38: finishing write tid 3 to trial16941936-38 2026-02-25T08:41:34.984 INFO:tasks.rados.rados.0.trial169.stdout:41: finishing write tid 3 to trial16941936-41 2026-02-25T08:41:34.984 INFO:tasks.rados.rados.0.trial169.stdout:38: finishing write tid 4 to trial16941936-38 2026-02-25T08:41:34.984 INFO:tasks.rados.rados.0.trial169.stdout:38: oid 38 updating version 0 to 9 2026-02-25T08:41:34.984 INFO:tasks.rados.rados.0.trial169.stdout:38: oid 38 version 9 is already newer than 9 2026-02-25T08:41:34.984 INFO:tasks.rados.rados.0.trial169.stdout:38: oid 38 version 9 is already newer than 7 2026-02-25T08:41:34.984 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 38 v 9 (ObjNum 37 snap 0 seq_num 37) dirty exists 2026-02-25T08:41:34.984 INFO:tasks.rados.rados.0.trial169.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2026-02-25T08:41:34.984 INFO:tasks.rados.rados.0.trial169.stdout:41: finishing write tid 4 to trial16941936-41 2026-02-25T08:41:34.984 INFO:tasks.rados.rados.0.trial169.stdout:41: oid 41 updating version 0 to 15 2026-02-25T08:41:34.984 INFO:tasks.rados.rados.0.trial169.stdout:41: oid 41 version 15 is already newer than 13 2026-02-25T08:41:34.984 INFO:tasks.rados.rados.0.trial169.stdout:41: oid 41 version 15 is already newer than 15 2026-02-25T08:41:34.984 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 41 v 15 (ObjNum 40 snap 0 seq_num 40) dirty exists 2026-02-25T08:41:34.984 INFO:tasks.rados.rados.0.trial169.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2026-02-25T08:41:34.985 INFO:tasks.rados.rados.0.trial169.stdout:35: finishing write tid 2 to trial16941936-35 2026-02-25T08:41:34.985 INFO:tasks.rados.rados.0.trial169.stdout:44: finishing write tid 1 to trial16941936-44 2026-02-25T08:41:34.985 INFO:tasks.rados.rados.0.trial169.stdout:43: finishing write tid 3 to trial16941936-43 2026-02-25T08:41:34.985 INFO:tasks.rados.rados.0.trial169.stdout:42: finishing write tid 1 to trial16941936-42 2026-02-25T08:41:34.985 INFO:tasks.rados.rados.0.trial169.stdout:43: finishing write tid 4 to trial16941936-43 2026-02-25T08:41:34.985 INFO:tasks.rados.rados.0.trial169.stdout:43: oid 43 updating version 0 to 7 2026-02-25T08:41:34.985 INFO:tasks.rados.rados.0.trial169.stdout:43: oid 43 updating version 7 to 9 2026-02-25T08:41:34.985 INFO:tasks.rados.rados.0.trial169.stdout:43: oid 43 version 9 is already newer than 9 2026-02-25T08:41:34.985 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 43 v 9 (ObjNum 42 snap 0 seq_num 42) dirty exists 2026-02-25T08:41:34.985 INFO:tasks.rados.rados.0.trial169.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2026-02-25T08:41:34.985 INFO:tasks.rados.rados.0.trial169.stdout:44: finishing write tid 2 to trial16941936-44 2026-02-25T08:41:34.985 INFO:tasks.rados.rados.0.trial169.stdout:35: finishing write tid 3 to trial16941936-35 2026-02-25T08:41:34.985 INFO:tasks.rados.rados.0.trial169.stdout:35: finishing write tid 4 to trial16941936-35 2026-02-25T08:41:34.986 INFO:tasks.rados.rados.0.trial169.stdout:35: oid 35 updating version 0 to 15 2026-02-25T08:41:34.986 INFO:tasks.rados.rados.0.trial169.stdout:35: oid 35 version 15 is already newer than 15 2026-02-25T08:41:34.986 INFO:tasks.rados.rados.0.trial169.stdout:35: oid 35 version 15 is already newer than 13 2026-02-25T08:41:34.986 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 35 v 15 (ObjNum 34 snap 0 seq_num 34) dirty exists 2026-02-25T08:41:34.986 INFO:tasks.rados.rados.0.trial169.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2026-02-25T08:41:34.986 INFO:tasks.rados.rados.0.trial169.stdout:44: finishing write tid 3 to trial16941936-44 2026-02-25T08:41:34.986 INFO:tasks.rados.rados.0.trial169.stdout:47: finishing write tid 1 to trial16941936-47 2026-02-25T08:41:34.986 INFO:tasks.rados.rados.0.trial169.stdout:44: finishing write tid 4 to trial16941936-44 2026-02-25T08:41:34.986 INFO:tasks.rados.rados.0.trial169.stdout:44: oid 44 updating version 0 to 13 2026-02-25T08:41:34.986 INFO:tasks.rados.rados.0.trial169.stdout:44: oid 44 updating version 13 to 15 2026-02-25T08:41:34.986 INFO:tasks.rados.rados.0.trial169.stdout:44: oid 44 version 15 is already newer than 15 2026-02-25T08:41:34.986 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 44 v 15 (ObjNum 43 snap 0 seq_num 43) dirty exists 2026-02-25T08:41:34.986 INFO:tasks.rados.rados.0.trial169.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2026-02-25T08:41:34.987 INFO:tasks.rados.rados.0.trial169.stdout:48: finishing write tid 1 to trial16941936-48 2026-02-25T08:41:34.987 INFO:tasks.rados.rados.0.trial169.stdout:35: done (15 left) 2026-02-25T08:41:34.987 INFO:tasks.rados.rados.0.trial169.stdout:38: done (14 left) 2026-02-25T08:41:34.987 INFO:tasks.rados.rados.0.trial169.stdout:41: done (13 left) 2026-02-25T08:41:34.987 INFO:tasks.rados.rados.0.trial169.stdout:43: done (12 left) 2026-02-25T08:41:34.987 INFO:tasks.rados.rados.0.trial169.stdout:44: done (11 left) 2026-02-25T08:41:34.987 INFO:tasks.rados.rados.0.trial169.stdout:54: read oid 10 snap -1 2026-02-25T08:41:34.987 INFO:tasks.rados.rados.0.trial169.stdout:54: expect (ObjNum 9 snap 0 seq_num 9) 2026-02-25T08:41:34.987 INFO:tasks.rados.rados.0.trial169.stdout:55: read oid 18 snap -1 2026-02-25T08:41:34.987 INFO:tasks.rados.rados.0.trial169.stdout:55: expect (ObjNum 17 snap 0 seq_num 17) 2026-02-25T08:41:34.987 INFO:tasks.rados.rados.0.trial169.stdout:56: read oid 1 snap -1 2026-02-25T08:41:34.987 INFO:tasks.rados.rados.0.trial169.stdout:56: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-25T08:41:34.987 INFO:tasks.rados.rados.0.trial169.stdout:57: read oid 44 snap -1 2026-02-25T08:41:34.988 INFO:tasks.rados.rados.0.trial169.stdout:57: expect (ObjNum 43 snap 0 seq_num 43) 2026-02-25T08:41:34.988 INFO:tasks.rados.rados.0.trial169.stdout:58: read oid 5 snap -1 2026-02-25T08:41:34.988 INFO:tasks.rados.rados.0.trial169.stdout:58: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:41:34.988 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:34.988 INFO:tasks.rados.rados.0.trial169.stdout:49: finishing write tid 1 to trial16941936-49 2026-02-25T08:41:34.988 INFO:tasks.rados.rados.0.trial169.stdout:40: finishing write tid 1 to trial16941936-40 2026-02-25T08:41:34.988 INFO:tasks.rados.rados.0.trial169.stdout:51: finishing write tid 1 to trial16941936-2 2026-02-25T08:41:34.989 INFO:tasks.rados.rados.0.trial169.stdout:42: finishing write tid 2 to trial16941936-42 2026-02-25T08:41:34.992 INFO:tasks.rados.rados.0.trial169.stdout:49: finishing write tid 2 to trial16941936-49 2026-02-25T08:41:35.001 INFO:tasks.rados.rados.0.trial169.stdout:47: finishing write tid 2 to trial16941936-47 2026-02-25T08:41:35.002 INFO:tasks.rados.rados.0.trial169.stdout:51: finishing write tid 2 to trial16941936-2 2026-02-25T08:41:35.006 INFO:tasks.rados.rados.0.trial169.stdout:55: expect (ObjNum 17 snap 0 seq_num 17) 2026-02-25T08:41:35.016 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:35.036 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:35.056 INFO:tasks.rados.rados.0.trial169.stdout:55: done (15 left) 2026-02-25T08:41:35.056 INFO:tasks.rados.rados.0.trial169.stdout:59: read oid 13 snap -1 2026-02-25T08:41:35.056 INFO:tasks.rados.rados.0.trial169.stdout:59: expect (ObjNum 12 snap 0 seq_num 12) 2026-02-25T08:41:35.056 INFO:tasks.rados.rados.0.trial169.stdout: waiting on 16 2026-02-25T08:41:35.056 INFO:tasks.rados.rados.0.trial169.stdout:58: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:41:35.056 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:35.076 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1599154 2026-02-25T08:41:35.097 INFO:tasks.rados.rados.0.trial169.stdout:49: finishing write tid 3 to trial16941936-49 2026-02-25T08:41:35.097 INFO:tasks.rados.rados.0.trial169.stdout:54: expect (ObjNum 9 snap 0 seq_num 9) 2026-02-25T08:41:35.112 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:35.144 INFO:tasks.rados.rados.0.trial169.stdout:53: expect (ObjNum 19 snap 0 seq_num 19) 2026-02-25T08:41:35.144 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:35.164 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:35.184 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:35.206 INFO:tasks.rados.rados.0.trial169.stdout:49: finishing write tid 4 to trial16941936-49 2026-02-25T08:41:35.206 INFO:tasks.rados.rados.0.trial169.stdout:49: oid 49 updating version 0 to 16 2026-02-25T08:41:35.206 INFO:tasks.rados.rados.0.trial169.stdout:49: oid 49 updating version 16 to 18 2026-02-25T08:41:35.206 INFO:tasks.rados.rados.0.trial169.stdout:49: oid 49 version 18 is already newer than 18 2026-02-25T08:41:35.207 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 49 v 18 (ObjNum 48 snap 0 seq_num 48) dirty exists 2026-02-25T08:41:35.207 INFO:tasks.rados.rados.0.trial169.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:41:35.207 INFO:tasks.rados.rados.0.trial169.stdout:51: finishing write tid 3 to trial16941936-2 2026-02-25T08:41:35.207 INFO:tasks.rados.rados.0.trial169.stdout:49: done (15 left) 2026-02-25T08:41:35.207 INFO:tasks.rados.rados.0.trial169.stdout:53: done (14 left) 2026-02-25T08:41:35.207 INFO:tasks.rados.rados.0.trial169.stdout:54: done (13 left) 2026-02-25T08:41:35.207 INFO:tasks.rados.rados.0.trial169.stdout:58: done (12 left) 2026-02-25T08:41:35.207 INFO:tasks.rados.rados.0.trial169.stdout:60: read oid 20 snap -1 2026-02-25T08:41:35.207 INFO:tasks.rados.rados.0.trial169.stdout:60: expect (ObjNum 19 snap 0 seq_num 19) 2026-02-25T08:41:35.207 INFO:tasks.rados.rados.0.trial169.stdout:61: write (excl) oid 28 current snap is 0 2026-02-25T08:41:35.208 INFO:tasks.rados.rados.0.trial169.stdout:51: finishing write tid 4 to trial16941936-2 2026-02-25T08:41:35.208 INFO:tasks.rados.rados.0.trial169.stdout:51: finishing write tid 5 to trial16941936-2 2026-02-25T08:41:35.208 INFO:tasks.rados.rados.0.trial169.stdout:51: oid 2 updating version 0 to 16 2026-02-25T08:41:35.208 INFO:tasks.rados.rados.0.trial169.stdout:51: oid 2 version 16 is already newer than 16 2026-02-25T08:41:35.208 INFO:tasks.rados.rados.0.trial169.stdout:51: oid 2 version 16 is already newer than 16 2026-02-25T08:41:35.208 INFO:tasks.rados.rados.0.trial169.stdout:51: oid 2 version 16 is already newer than 13 2026-02-25T08:41:35.208 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 2 v 16 (ObjNum 50 snap 0 seq_num 50) dirty exists 2026-02-25T08:41:35.208 INFO:tasks.rados.rados.0.trial169.stdout:51: left oid 2 (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:41:35.208 INFO:tasks.rados.rados.0.trial169.stdout:40: finishing write tid 2 to trial16941936-40 2026-02-25T08:41:35.208 INFO:tasks.rados.rados.0.trial169.stdout:57: expect (ObjNum 43 snap 0 seq_num 43) 2026-02-25T08:41:35.265 INFO:tasks.rados.rados.0.trial169.stdout:40: finishing write tid 3 to trial16941936-40 2026-02-25T08:41:35.266 INFO:tasks.rados.rados.0.trial169.stdout:56: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-25T08:41:35.271 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1612440 2026-02-25T08:41:35.274 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:35.309 INFO:tasks.rados.rados.0.trial169.stdout:40: finishing write tid 4 to trial16941936-40 2026-02-25T08:41:35.309 INFO:tasks.rados.rados.0.trial169.stdout:40: oid 40 updating version 0 to 18 2026-02-25T08:41:35.309 INFO:tasks.rados.rados.0.trial169.stdout:40: oid 40 version 18 is already newer than 16 2026-02-25T08:41:35.309 INFO:tasks.rados.rados.0.trial169.stdout:40: oid 40 version 18 is already newer than 18 2026-02-25T08:41:35.309 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 40 v 18 (ObjNum 39 snap 0 seq_num 39) dirty exists 2026-02-25T08:41:35.309 INFO:tasks.rados.rados.0.trial169.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2026-02-25T08:41:35.310 INFO:tasks.rados.rados.0.trial169.stdout:47: finishing write tid 3 to trial16941936-47 2026-02-25T08:41:35.310 INFO:tasks.rados.rados.0.trial169.stdout:61: seq_num 52 ranges {473964=611066,1826593=546335,2778819=95145} 2026-02-25T08:41:35.313 INFO:tasks.rados.rados.0.trial169.stdout:61: writing trial16941936-28 from 473964 to 1085030 tid 1 2026-02-25T08:41:35.318 INFO:tasks.rados.rados.0.trial169.stdout:61: writing trial16941936-28 from 1826593 to 2372928 tid 2 2026-02-25T08:41:35.321 INFO:tasks.rados.rados.0.trial169.stdout:61: writing trial16941936-28 from 2778819 to 2873964 tid 3 2026-02-25T08:41:35.321 INFO:tasks.rados.rados.0.trial169.stdout:40: done (13 left) 2026-02-25T08:41:35.321 INFO:tasks.rados.rados.0.trial169.stdout:51: done (12 left) 2026-02-25T08:41:35.321 INFO:tasks.rados.rados.0.trial169.stdout:56: done (11 left) 2026-02-25T08:41:35.322 INFO:tasks.rados.rados.0.trial169.stdout:57: done (10 left) 2026-02-25T08:41:35.322 INFO:tasks.rados.rados.0.trial169.stdout:62: snap_create 2026-02-25T08:41:35.322 INFO:tasks.rados.rados.0.trial169.stdout:48: finishing write tid 2 to trial16941936-48 2026-02-25T08:41:35.322 INFO:tasks.rados.rados.0.trial169.stdout:47: finishing write tid 4 to trial16941936-47 2026-02-25T08:41:35.322 INFO:tasks.rados.rados.0.trial169.stdout:47: oid 47 updating version 0 to 10 2026-02-25T08:41:35.322 INFO:tasks.rados.rados.0.trial169.stdout:47: oid 47 updating version 10 to 12 2026-02-25T08:41:35.322 INFO:tasks.rados.rados.0.trial169.stdout:47: oid 47 version 12 is already newer than 12 2026-02-25T08:41:35.322 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2026-02-25T08:41:35.322 INFO:tasks.rados.rados.0.trial169.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2026-02-25T08:41:35.322 INFO:tasks.rados.rados.0.trial169.stdout:48: finishing write tid 3 to trial16941936-48 2026-02-25T08:41:35.322 INFO:tasks.rados.rados.0.trial169.stdout:42: finishing write tid 3 to trial16941936-42 2026-02-25T08:41:35.323 INFO:tasks.rados.rados.0.trial169.stdout:50: finishing write tid 1 to trial16941936-50 2026-02-25T08:41:35.323 INFO:tasks.rados.rados.0.trial169.stdout:48: finishing write tid 4 to trial16941936-48 2026-02-25T08:41:35.323 INFO:tasks.rados.rados.0.trial169.stdout:48: oid 48 updating version 0 to 3 2026-02-25T08:41:35.323 INFO:tasks.rados.rados.0.trial169.stdout:48: oid 48 version 3 is already newer than 1 2026-02-25T08:41:35.323 INFO:tasks.rados.rados.0.trial169.stdout:48: oid 48 version 3 is already newer than 3 2026-02-25T08:41:35.323 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 48 v 3 (ObjNum 47 snap 0 seq_num 47) dirty exists 2026-02-25T08:41:35.323 INFO:tasks.rados.rados.0.trial169.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:35.323 INFO:tasks.rados.rados.0.trial169.stdout:42: finishing write tid 4 to trial16941936-42 2026-02-25T08:41:35.323 INFO:tasks.rados.rados.0.trial169.stdout:42: oid 42 updating version 0 to 7 2026-02-25T08:41:35.323 INFO:tasks.rados.rados.0.trial169.stdout:42: oid 42 updating version 7 to 9 2026-02-25T08:41:35.323 INFO:tasks.rados.rados.0.trial169.stdout:42: oid 42 version 9 is already newer than 9 2026-02-25T08:41:35.324 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2026-02-25T08:41:35.324 INFO:tasks.rados.rados.0.trial169.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2026-02-25T08:41:35.324 INFO:tasks.rados.rados.0.trial169.stdout:45: finishing write tid 1 to trial16941936-45 2026-02-25T08:41:35.324 INFO:tasks.rados.rados.0.trial169.stdout:50: finishing write tid 2 to trial16941936-50 2026-02-25T08:41:35.324 INFO:tasks.rados.rados.0.trial169.stdout:59: expect (ObjNum 12 snap 0 seq_num 12) 2026-02-25T08:41:35.339 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 732770 2026-02-25T08:41:35.366 INFO:tasks.rados.rados.0.trial169.stdout:45: finishing write tid 2 to trial16941936-45 2026-02-25T08:41:35.367 INFO:tasks.rados.rados.0.trial169.stdout:45: finishing write tid 3 to trial16941936-45 2026-02-25T08:41:35.367 INFO:tasks.rados.rados.0.trial169.stdout:45: finishing write tid 4 to trial16941936-45 2026-02-25T08:41:35.367 INFO:tasks.rados.rados.0.trial169.stdout:45: oid 45 updating version 0 to 12 2026-02-25T08:41:35.367 INFO:tasks.rados.rados.0.trial169.stdout:45: oid 45 version 12 is already newer than 12 2026-02-25T08:41:35.367 INFO:tasks.rados.rados.0.trial169.stdout:45: oid 45 version 12 is already newer than 10 2026-02-25T08:41:35.367 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 45 v 12 (ObjNum 44 snap 0 seq_num 44) dirty exists 2026-02-25T08:41:35.367 INFO:tasks.rados.rados.0.trial169.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2026-02-25T08:41:35.367 INFO:tasks.rados.rados.0.trial169.stdout:50: finishing write tid 3 to trial16941936-50 2026-02-25T08:41:35.367 INFO:tasks.rados.rados.0.trial169.stdout:50: finishing write tid 4 to trial16941936-50 2026-02-25T08:41:35.367 INFO:tasks.rados.rados.0.trial169.stdout:50: oid 50 updating version 0 to 21 2026-02-25T08:41:35.368 INFO:tasks.rados.rados.0.trial169.stdout:50: oid 50 version 21 is already newer than 21 2026-02-25T08:41:35.368 INFO:tasks.rados.rados.0.trial169.stdout:50: oid 50 version 21 is already newer than 19 2026-02-25T08:41:35.368 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 50 v 21 (ObjNum 49 snap 0 seq_num 49) dirty exists 2026-02-25T08:41:35.368 INFO:tasks.rados.rados.0.trial169.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2026-02-25T08:41:35.368 INFO:tasks.rados.rados.0.trial169.stdout:46: finishing write tid 1 to trial16941936-46 2026-02-25T08:41:35.368 INFO:tasks.rados.rados.0.trial169.stdout:46: finishing write tid 2 to trial16941936-46 2026-02-25T08:41:35.368 INFO:tasks.rados.rados.0.trial169.stdout:46: finishing write tid 3 to trial16941936-46 2026-02-25T08:41:35.368 INFO:tasks.rados.rados.0.trial169.stdout:46: finishing write tid 4 to trial16941936-46 2026-02-25T08:41:35.368 INFO:tasks.rados.rados.0.trial169.stdout:46: oid 46 updating version 0 to 13 2026-02-25T08:41:35.368 INFO:tasks.rados.rados.0.trial169.stdout:46: oid 46 updating version 13 to 15 2026-02-25T08:41:35.369 INFO:tasks.rados.rados.0.trial169.stdout:46: oid 46 version 15 is already newer than 15 2026-02-25T08:41:35.369 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 46 v 15 (ObjNum 45 snap 0 seq_num 45) dirty exists 2026-02-25T08:41:35.369 INFO:tasks.rados.rados.0.trial169.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2026-02-25T08:41:35.369 INFO:tasks.rados.rados.0.trial169.stdout:52: finishing write tid 1 to trial16941936-14 2026-02-25T08:41:35.369 INFO:tasks.rados.rados.0.trial169.stdout:52: finishing write tid 2 to trial16941936-14 2026-02-25T08:41:35.369 INFO:tasks.rados.rados.0.trial169.stdout:52: finishing write tid 3 to trial16941936-14 2026-02-25T08:41:35.369 INFO:tasks.rados.rados.0.trial169.stdout:52: finishing write tid 4 to trial16941936-14 2026-02-25T08:41:35.369 INFO:tasks.rados.rados.0.trial169.stdout:52: oid 14 updating version 0 to 18 2026-02-25T08:41:35.369 INFO:tasks.rados.rados.0.trial169.stdout:52: oid 14 version 18 is already newer than 16 2026-02-25T08:41:35.369 INFO:tasks.rados.rados.0.trial169.stdout:52: oid 14 version 18 is already newer than 18 2026-02-25T08:41:35.369 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 14 v 18 (ObjNum 51 snap 0 seq_num 51) dirty exists 2026-02-25T08:41:35.370 INFO:tasks.rados.rados.0.trial169.stdout:52: left oid 14 (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:41:35.370 INFO:tasks.rados.rados.0.trial169.stdout:60: expect (ObjNum 19 snap 0 seq_num 19) 2026-02-25T08:41:35.406 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:35.426 INFO:tasks.rados.rados.0.trial169.stdout:61: finishing write tid 1 to trial16941936-28 2026-02-25T08:41:35.426 INFO:tasks.rados.rados.0.trial169.stdout:61: finishing write tid 2 to trial16941936-28 2026-02-25T08:41:35.426 INFO:tasks.rados.rados.0.trial169.stdout:61: finishing write tid 3 to trial16941936-28 2026-02-25T08:41:35.426 INFO:tasks.rados.rados.0.trial169.stdout:61: finishing write tid 4 to trial16941936-28 2026-02-25T08:41:35.426 INFO:tasks.rados.rados.0.trial169.stdout:61: finishing write tid 5 to trial16941936-28 2026-02-25T08:41:35.427 INFO:tasks.rados.rados.0.trial169.stdout:61: oid 28 updating version 0 to 10 2026-02-25T08:41:35.427 INFO:tasks.rados.rados.0.trial169.stdout:61: oid 28 version 10 is already newer than 10 2026-02-25T08:41:35.427 INFO:tasks.rados.rados.0.trial169.stdout:61: oid 28 version 10 is already newer than 10 2026-02-25T08:41:35.427 INFO:tasks.rados.rados.0.trial169.stdout:61: oid 28 version 10 is already newer than 7 2026-02-25T08:41:35.427 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 28 v 10 (ObjNum 52 snap 0 seq_num 52) dirty exists 2026-02-25T08:41:35.427 INFO:tasks.rados.rados.0.trial169.stdout:61: left oid 28 (ObjNum 52 snap 0 seq_num 52) 2026-02-25T08:41:35.507 INFO:tasks.rados.rados.0.trial169.stdout:42: done (10 left) 2026-02-25T08:41:35.507 INFO:tasks.rados.rados.0.trial169.stdout:45: done (9 left) 2026-02-25T08:41:35.507 INFO:tasks.rados.rados.0.trial169.stdout:46: done (8 left) 2026-02-25T08:41:35.507 INFO:tasks.rados.rados.0.trial169.stdout:47: done (7 left) 2026-02-25T08:41:35.508 INFO:tasks.rados.rados.0.trial169.stdout:48: done (6 left) 2026-02-25T08:41:35.508 INFO:tasks.rados.rados.0.trial169.stdout:50: done (5 left) 2026-02-25T08:41:35.508 INFO:tasks.rados.rados.0.trial169.stdout:52: done (4 left) 2026-02-25T08:41:35.508 INFO:tasks.rados.rados.0.trial169.stdout:59: done (3 left) 2026-02-25T08:41:35.508 INFO:tasks.rados.rados.0.trial169.stdout:60: done (2 left) 2026-02-25T08:41:35.508 INFO:tasks.rados.rados.0.trial169.stdout:61: done (1 left) 2026-02-25T08:41:35.508 INFO:tasks.rados.rados.0.trial169.stdout:62: done (0 left) 2026-02-25T08:41:35.508 INFO:tasks.rados.rados.0.trial169.stdout:63: write oid 45 current snap is 1 2026-02-25T08:41:35.508 INFO:tasks.rados.rados.0.trial169.stdout:63: seq_num 53 ranges {490771=686315,1885075=406346} 2026-02-25T08:41:35.511 INFO:tasks.rados.rados.0.trial169.stdout:63: writing trial16941936-45 from 490771 to 1177086 tid 1 2026-02-25T08:41:35.516 INFO:tasks.rados.rados.0.trial169.stdout:63: writing trial16941936-45 from 1885075 to 2291421 tid 2 2026-02-25T08:41:35.516 INFO:tasks.rados.rados.0.trial169.stdout:64: read oid 16 snap -1 2026-02-25T08:41:35.517 INFO:tasks.rados.rados.0.trial169.stdout:64: expect (ObjNum 15 snap 0 seq_num 15) 2026-02-25T08:41:35.517 INFO:tasks.rados.rados.0.trial169.stdout:65: write oid 44 current snap is 1 2026-02-25T08:41:35.517 INFO:tasks.rados.rados.0.trial169.stdout:65: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2026-02-25T08:41:35.520 INFO:tasks.rados.rados.0.trial169.stdout:65: writing trial16941936-44 from 507578 to 985495 tid 1 2026-02-25T08:41:35.524 INFO:tasks.rados.rados.0.trial169.stdout:65: writing trial16941936-44 from 1543557 to 2093561 tid 2 2026-02-25T08:41:35.527 INFO:tasks.rados.rados.0.trial169.stdout:65: writing trial16941936-44 from 2833665 to 2907578 tid 3 2026-02-25T08:41:35.527 INFO:tasks.rados.rados.0.trial169.stdout:66: read oid 48 snap -1 2026-02-25T08:41:35.527 INFO:tasks.rados.rados.0.trial169.stdout:66: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:35.528 INFO:tasks.rados.rados.0.trial169.stdout:67: delete oid 31 current snap is 1 2026-02-25T08:41:35.535 INFO:tasks.rados.rados.0.trial169.stdout:64: expect (ObjNum 15 snap 0 seq_num 15) 2026-02-25T08:41:35.536 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:35.594 INFO:tasks.rados.rados.0.trial169.stdout:64: done (4 left) 2026-02-25T08:41:35.595 INFO:tasks.rados.rados.0.trial169.stdout:67: done (3 left) 2026-02-25T08:41:35.595 INFO:tasks.rados.rados.0.trial169.stdout:68: snap_remove snap 0 2026-02-25T08:41:35.871 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:41:35.947 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:35.945+0000 7f374363d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:35.950 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:35.948+0000 7f374363d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:36.513 INFO:tasks.rados.rados.0.trial169.stdout:68: done (3 left) 2026-02-25T08:41:36.513 INFO:tasks.rados.rados.0.trial169.stdout:69: read oid 46 snap -1 2026-02-25T08:41:36.513 INFO:tasks.rados.rados.0.trial169.stdout:69: expect (ObjNum 45 snap 0 seq_num 45) 2026-02-25T08:41:36.513 INFO:tasks.rados.rados.0.trial169.stdout:70: read oid 34 snap -1 2026-02-25T08:41:36.513 INFO:tasks.rados.rados.0.trial169.stdout:70: expect (ObjNum 33 snap 0 seq_num 33) 2026-02-25T08:41:36.513 INFO:tasks.rados.rados.0.trial169.stdout:71: read oid 19 snap -1 2026-02-25T08:41:36.513 INFO:tasks.rados.rados.0.trial169.stdout:71: expect (ObjNum 18 snap 0 seq_num 18) 2026-02-25T08:41:36.513 INFO:tasks.rados.rados.0.trial169.stdout:72: read oid 27 snap -1 2026-02-25T08:41:36.513 INFO:tasks.rados.rados.0.trial169.stdout:72: expect (ObjNum 26 snap 0 seq_num 26) 2026-02-25T08:41:36.514 INFO:tasks.rados.rados.0.trial169.stdout:73: read oid 4 snap -1 2026-02-25T08:41:36.514 INFO:tasks.rados.rados.0.trial169.stdout:73: expect (ObjNum 3 snap 0 seq_num 3) 2026-02-25T08:41:36.514 INFO:tasks.rados.rados.0.trial169.stdout:74: write oid 24 current snap is 1 2026-02-25T08:41:36.514 INFO:tasks.rados.rados.0.trial169.stdout:74: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2026-02-25T08:41:36.516 INFO:tasks.rados.rados.0.trial169.stdout:74: writing trial16941936-24 from 541192 to 1169607 tid 1 2026-02-25T08:41:36.520 INFO:tasks.rados.rados.0.trial169.stdout:74: writing trial16941936-24 from 1660521 to 2214194 tid 2 2026-02-25T08:41:36.524 INFO:tasks.rados.rados.0.trial169.stdout:74: writing trial16941936-24 from 2888511 to 2941192 tid 3 2026-02-25T08:41:36.524 INFO:tasks.rados.rados.0.trial169.stdout:75: read oid 5 snap -1 2026-02-25T08:41:36.524 INFO:tasks.rados.rados.0.trial169.stdout:75: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:41:36.524 INFO:tasks.rados.rados.0.trial169.stdout:76: read oid 1 snap -1 2026-02-25T08:41:36.524 INFO:tasks.rados.rados.0.trial169.stdout:76: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-25T08:41:36.524 INFO:tasks.rados.rados.0.trial169.stdout:77: write (excl) oid 6 current snap is 1 2026-02-25T08:41:36.524 INFO:tasks.rados.rados.0.trial169.stdout:63: finishing write tid 1 to trial16941936-45 2026-02-25T08:41:36.524 INFO:tasks.rados.rados.0.trial169.stdout:77: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2026-02-25T08:41:36.528 INFO:tasks.rados.rados.0.trial169.stdout:77: writing trial16941936-6 from 557999 to 1261663 tid 1 2026-02-25T08:41:36.531 INFO:tasks.rados.rados.0.trial169.stdout:77: writing trial16941936-6 from 1719003 to 2132687 tid 2 2026-02-25T08:41:36.535 INFO:tasks.rados.rados.0.trial169.stdout:77: writing trial16941936-6 from 2632287 to 2957999 tid 3 2026-02-25T08:41:36.535 INFO:tasks.rados.rados.0.trial169.stdout:78: write oid 9 current snap is 1 2026-02-25T08:41:36.535 INFO:tasks.rados.rados.0.trial169.stdout:78: seq_num 58 ranges {574806=778913,2061132=557342} 2026-02-25T08:41:36.540 INFO:tasks.rados.rados.0.trial169.stdout:78: writing trial16941936-9 from 574806 to 1353719 tid 1 2026-02-25T08:41:36.545 INFO:tasks.rados.rados.0.trial169.stdout:78: writing trial16941936-9 from 2061132 to 2618474 tid 2 2026-02-25T08:41:36.545 INFO:tasks.rados.rados.0.trial169.stdout:79: delete oid 11 current snap is 1 2026-02-25T08:41:36.545 INFO:tasks.rados.rados.0.trial169.stdout:63: finishing write tid 2 to trial16941936-45 2026-02-25T08:41:36.546 INFO:tasks.rados.rados.0.trial169.stdout:63: finishing write tid 3 to trial16941936-45 2026-02-25T08:41:36.546 INFO:tasks.rados.rados.0.trial169.stdout:63: finishing write tid 4 to trial16941936-45 2026-02-25T08:41:36.546 INFO:tasks.rados.rados.0.trial169.stdout:63: oid 45 updating version 0 to 22 2026-02-25T08:41:36.546 INFO:tasks.rados.rados.0.trial169.stdout:63: oid 45 version 22 is already newer than 22 2026-02-25T08:41:36.546 INFO:tasks.rados.rados.0.trial169.stdout:63: oid 45 version 22 is already newer than 22 2026-02-25T08:41:36.546 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 45 v 22 (ObjNum 53 snap 1 seq_num 53) dirty exists 2026-02-25T08:41:36.546 INFO:tasks.rados.rados.0.trial169.stdout:63: left oid 45 (ObjNum 53 snap 1 seq_num 53) 2026-02-25T08:41:36.546 INFO:tasks.rados.rados.0.trial169.stdout:65: finishing write tid 1 to trial16941936-44 2026-02-25T08:41:36.546 INFO:tasks.rados.rados.0.trial169.stdout:66: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:36.546 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:36.573 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1154049 2026-02-25T08:41:36.591 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 999652 2026-02-25T08:41:36.606 INFO:tasks.rados.rados.0.trial169.stdout:65: finishing write tid 2 to trial16941936-44 2026-02-25T08:41:36.606 INFO:tasks.rados.rados.0.trial169.stdout:65: finishing write tid 3 to trial16941936-44 2026-02-25T08:41:36.606 INFO:tasks.rados.rados.0.trial169.stdout:65: finishing write tid 4 to trial16941936-44 2026-02-25T08:41:36.606 INFO:tasks.rados.rados.0.trial169.stdout:65: finishing write tid 5 to trial16941936-44 2026-02-25T08:41:36.607 INFO:tasks.rados.rados.0.trial169.stdout:65: oid 44 updating version 0 to 20 2026-02-25T08:41:36.607 INFO:tasks.rados.rados.0.trial169.stdout:65: oid 44 version 20 is already newer than 20 2026-02-25T08:41:36.607 INFO:tasks.rados.rados.0.trial169.stdout:65: oid 44 version 20 is already newer than 20 2026-02-25T08:41:36.607 INFO:tasks.rados.rados.0.trial169.stdout:65: oid 44 version 20 is already newer than 20 2026-02-25T08:41:36.607 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 44 v 20 (ObjNum 54 snap 1 seq_num 54) dirty exists 2026-02-25T08:41:36.607 INFO:tasks.rados.rados.0.trial169.stdout:65: left oid 44 (ObjNum 54 snap 1 seq_num 54) 2026-02-25T08:41:36.607 INFO:tasks.rados.rados.0.trial169.stdout:63: done (13 left) 2026-02-25T08:41:36.607 INFO:tasks.rados.rados.0.trial169.stdout:65: done (12 left) 2026-02-25T08:41:36.607 INFO:tasks.rados.rados.0.trial169.stdout:66: done (11 left) 2026-02-25T08:41:36.607 INFO:tasks.rados.rados.0.trial169.stdout:79: done (10 left) 2026-02-25T08:41:36.607 INFO:tasks.rados.rados.0.trial169.stdout:80: snap_create 2026-02-25T08:41:36.607 INFO:tasks.rados.rados.0.trial169.stdout:75: expect (ObjNum 4 snap 0 seq_num 4) 2026-02-25T08:41:36.640 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1615975 2026-02-25T08:41:36.648 INFO:tasks.rados.rados.0.trial169.stdout:72: expect (ObjNum 26 snap 0 seq_num 26) 2026-02-25T08:41:36.686 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2303335 2026-02-25T08:41:36.687 INFO:tasks.rados.rados.0.trial169.stdout:69: expect (ObjNum 45 snap 0 seq_num 45) 2026-02-25T08:41:36.709 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 562013 2026-02-25T08:41:36.729 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 185258 2026-02-25T08:41:36.751 INFO:tasks.rados.rados.0.trial169.stdout:74: finishing write tid 1 to trial16941936-24 2026-02-25T08:41:36.751 INFO:tasks.rados.rados.0.trial169.stdout:73: expect (ObjNum 3 snap 0 seq_num 3) 2026-02-25T08:41:36.753 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1309213 2026-02-25T08:41:36.762 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:36.779 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:36.797 INFO:tasks.rados.rados.0.trial169.stdout:70: expect (ObjNum 33 snap 0 seq_num 33) 2026-02-25T08:41:36.813 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1389493 2026-02-25T08:41:36.826 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:36.848 INFO:tasks.rados.rados.0.trial169.stdout:71: expect (ObjNum 18 snap 0 seq_num 18) 2026-02-25T08:41:36.848 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:36.908 INFO:tasks.rados.rados.0.trial169.stdout:76: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-25T08:41:36.948 INFO:tasks.rados.rados.0.trial169.stdout:74: finishing write tid 2 to trial16941936-24 2026-02-25T08:41:36.948 INFO:tasks.rados.rados.0.trial169.stdout:77: finishing write tid 1 to trial16941936-6 2026-02-25T08:41:36.948 INFO:tasks.rados.rados.0.trial169.stdout:74: finishing write tid 3 to trial16941936-24 2026-02-25T08:41:36.948 INFO:tasks.rados.rados.0.trial169.stdout:78: finishing write tid 1 to trial16941936-9 2026-02-25T08:41:36.948 INFO:tasks.rados.rados.0.trial169.stdout:74: finishing write tid 4 to trial16941936-24 2026-02-25T08:41:36.948 INFO:tasks.rados.rados.0.trial169.stdout:74: finishing write tid 5 to trial16941936-24 2026-02-25T08:41:36.948 INFO:tasks.rados.rados.0.trial169.stdout:74: oid 24 updating version 0 to 10 2026-02-25T08:41:36.948 INFO:tasks.rados.rados.0.trial169.stdout:74: oid 24 version 10 is already newer than 10 2026-02-25T08:41:36.948 INFO:tasks.rados.rados.0.trial169.stdout:74: oid 24 version 10 is already newer than 10 2026-02-25T08:41:36.949 INFO:tasks.rados.rados.0.trial169.stdout:74: oid 24 version 10 is already newer than 10 2026-02-25T08:41:36.949 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 24 v 10 (ObjNum 56 snap 1 seq_num 56) dirty exists 2026-02-25T08:41:36.949 INFO:tasks.rados.rados.0.trial169.stdout:74: left oid 24 (ObjNum 56 snap 1 seq_num 56) 2026-02-25T08:41:36.949 INFO:tasks.rados.rados.0.trial169.stdout:78: finishing write tid 2 to trial16941936-9 2026-02-25T08:41:36.949 INFO:tasks.rados.rados.0.trial169.stdout:78: finishing write tid 3 to trial16941936-9 2026-02-25T08:41:36.949 INFO:tasks.rados.rados.0.trial169.stdout:78: finishing write tid 4 to trial16941936-9 2026-02-25T08:41:36.949 INFO:tasks.rados.rados.0.trial169.stdout:78: oid 9 updating version 0 to 15 2026-02-25T08:41:36.949 INFO:tasks.rados.rados.0.trial169.stdout:78: oid 9 version 15 is already newer than 15 2026-02-25T08:41:36.949 INFO:tasks.rados.rados.0.trial169.stdout:78: oid 9 version 15 is already newer than 15 2026-02-25T08:41:36.949 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 9 v 15 (ObjNum 58 snap 1 seq_num 58) dirty exists 2026-02-25T08:41:36.949 INFO:tasks.rados.rados.0.trial169.stdout:78: left oid 9 (ObjNum 58 snap 1 seq_num 58) 2026-02-25T08:41:36.949 INFO:tasks.rados.rados.0.trial169.stdout:77: finishing write tid 2 to trial16941936-6 2026-02-25T08:41:36.950 INFO:tasks.rados.rados.0.trial169.stdout:77: finishing write tid 3 to trial16941936-6 2026-02-25T08:41:36.950 INFO:tasks.rados.rados.0.trial169.stdout:77: finishing write tid 4 to trial16941936-6 2026-02-25T08:41:36.950 INFO:tasks.rados.rados.0.trial169.stdout:77: finishing write tid 5 to trial16941936-6 2026-02-25T08:41:36.950 INFO:tasks.rados.rados.0.trial169.stdout:77: oid 6 updating version 0 to 10 2026-02-25T08:41:36.950 INFO:tasks.rados.rados.0.trial169.stdout:77: oid 6 updating version 10 to 13 2026-02-25T08:41:36.950 INFO:tasks.rados.rados.0.trial169.stdout:77: oid 6 version 13 is already newer than 13 2026-02-25T08:41:36.950 INFO:tasks.rados.rados.0.trial169.stdout:77: oid 6 version 13 is already newer than 13 2026-02-25T08:41:36.950 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 6 v 13 (ObjNum 57 snap 1 seq_num 57) dirty exists 2026-02-25T08:41:36.950 INFO:tasks.rados.rados.0.trial169.stdout:77: left oid 6 (ObjNum 57 snap 1 seq_num 57) 2026-02-25T08:41:37.480 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:37.514 INFO:tasks.rados.rados.0.trial169.stdout:69: done (10 left) 2026-02-25T08:41:37.514 INFO:tasks.rados.rados.0.trial169.stdout:70: done (9 left) 2026-02-25T08:41:37.514 INFO:tasks.rados.rados.0.trial169.stdout:71: done (8 left) 2026-02-25T08:41:37.515 INFO:tasks.rados.rados.0.trial169.stdout:72: done (7 left) 2026-02-25T08:41:37.515 INFO:tasks.rados.rados.0.trial169.stdout:73: done (6 left) 2026-02-25T08:41:37.515 INFO:tasks.rados.rados.0.trial169.stdout:74: done (5 left) 2026-02-25T08:41:37.515 INFO:tasks.rados.rados.0.trial169.stdout:75: done (4 left) 2026-02-25T08:41:37.515 INFO:tasks.rados.rados.0.trial169.stdout:76: done (3 left) 2026-02-25T08:41:37.515 INFO:tasks.rados.rados.0.trial169.stdout:77: done (2 left) 2026-02-25T08:41:37.515 INFO:tasks.rados.rados.0.trial169.stdout:78: done (1 left) 2026-02-25T08:41:37.515 INFO:tasks.rados.rados.0.trial169.stdout:80: done (0 left) 2026-02-25T08:41:37.515 INFO:tasks.rados.rados.0.trial169.stdout:81: delete oid 42 current snap is 2 2026-02-25T08:41:37.550 INFO:tasks.rados.rados.0.trial169.stdout:81: done (0 left) 2026-02-25T08:41:37.550 INFO:tasks.rados.rados.0.trial169.stdout:82: read oid 48 snap 1 2026-02-25T08:41:37.550 INFO:tasks.rados.rados.0.trial169.stdout:82: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:37.550 INFO:tasks.rados.rados.0.trial169.stdout:83: delete oid 5 current snap is 2 2026-02-25T08:41:37.562 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:37.558+0000 7f0c1397e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:37.565 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:37.563+0000 7f0c1397e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:37.565 INFO:tasks.rados.rados.0.trial169.stdout:82: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:37.593 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:37.616 INFO:tasks.rados.rados.0.trial169.stdout:82: done (1 left) 2026-02-25T08:41:37.617 INFO:tasks.rados.rados.0.trial169.stdout:83: done (0 left) 2026-02-25T08:41:37.617 INFO:tasks.rados.rados.0.trial169.stdout:84: snap_create 2026-02-25T08:41:37.678 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:37.679 INFO:teuthology.orchestra.run.trial169.stderr:dumped all 2026-02-25T08:41:37.686 INFO:teuthology.orchestra.run.trial169.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-02-25T08:41:37.511459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":115512202,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":221,"num_read_kb":55140,"num_write":315,"num_write_kb":64344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":84089168,"kb_used_data":84089168,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125625968,"statfs":{"total":214748299264,"available":128640991232,"internally_reserved":0,"allocated":86107308032,"data_stored":86107308032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":114921815,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":168,"num_read_kb":55094,"num_write":234,"num_write_kb":63236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011624"},"pg_stats":[{"pgid":"3.f","version":"20'9","reported_seq":45,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.538422+0000","last_change":"2026-02-25T08:41:36.528428+0000","last_active":"2026-02-25T08:41:36.538422+0000","last_peered":"2026-02-25T08:41:36.538422+0000","last_clean":"2026-02-25T08:41:36.538422+0000","last_became_active":"2026-02-25T08:41:31.577339+0000","last_became_peered":"2026-02-25T08:41:31.577339+0000","last_unstale":"2026-02-25T08:41:36.538422+0000","last_undegraded":"2026-02-25T08:41:36.538422+0000","last_fullsized":"2026-02-25T08:41:36.538422+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6386561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":7817,"num_write":12,"num_write_kb":3115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.522682+0000","last_change":"2026-02-25T08:41:36.528396+0000","last_active":"2026-02-25T08:41:36.522682+0000","last_peered":"2026-02-25T08:41:36.522682+0000","last_clean":"2026-02-25T08:41:36.522682+0000","last_became_active":"2026-02-25T08:41:31.567579+0000","last_became_peered":"2026-02-25T08:41:31.567579+0000","last_unstale":"2026-02-25T08:41:36.522682+0000","last_undegraded":"2026-02-25T08:41:36.522682+0000","last_fullsized":"2026-02-25T08:41:36.522682+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'3","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.522858+0000","last_change":"2026-02-25T08:41:36.527143+0000","last_active":"2026-02-25T08:41:36.522858+0000","last_peered":"2026-02-25T08:41:36.522858+0000","last_clean":"2026-02-25T08:41:36.522858+0000","last_became_active":"2026-02-25T08:41:31.550562+0000","last_became_peered":"2026-02-25T08:41:31.550562+0000","last_unstale":"2026-02-25T08:41:36.522858+0000","last_undegraded":"2026-02-25T08:41:36.522858+0000","last_fullsized":"2026-02-25T08:41:36.522858+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2789929,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6073,"num_write":4,"num_write_kb":1343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"20'16","reported_seq":46,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.524807+0000","last_change":"2026-02-25T08:41:36.533516+0000","last_active":"2026-02-25T08:41:36.524807+0000","last_peered":"2026-02-25T08:41:36.524807+0000","last_clean":"2026-02-25T08:41:36.524807+0000","last_became_active":"2026-02-25T08:41:31.571602+0000","last_became_peered":"2026-02-25T08:41:31.571602+0000","last_unstale":"2026-02-25T08:41:36.524807+0000","last_undegraded":"2026-02-25T08:41:36.524807+0000","last_fullsized":"2026-02-25T08:41:36.524807+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9966419,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":21,"num_write_kb":5481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"20'9","reported_seq":33,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-25T08:41:35.515751+0000","last_change":"2026-02-25T08:41:31.551969+0000","last_active":"2026-02-25T08:41:35.515751+0000","last_peered":"2026-02-25T08:41:35.515751+0000","last_clean":"2026-02-25T08:41:35.515751+0000","last_became_active":"2026-02-25T08:41:31.546602+0000","last_became_peered":"2026-02-25T08:41:31.546602+0000","last_unstale":"2026-02-25T08:41:35.515751+0000","last_undegraded":"2026-02-25T08:41:35.515751+0000","last_fullsized":"2026-02-25T08:41:35.515751+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7226911,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"21'20","reported_seq":52,"reported_epoch":22,"state":"active+clean+snaptrim","last_fresh":"2026-02-25T08:41:36.525531+0000","last_change":"2026-02-25T08:41:36.525384+0000","last_active":"2026-02-25T08:41:36.525531+0000","last_peered":"2026-02-25T08:41:36.525531+0000","last_clean":"2026-02-25T08:41:36.525531+0000","last_became_active":"2026-02-25T08:41:31.576273+0000","last_became_peered":"2026-02-25T08:41:31.576273+0000","last_unstale":"2026-02-25T08:41:36.525531+0000","last_undegraded":"2026-02-25T08:41:36.525531+0000","last_fullsized":"2026-02-25T08:41:36.525531+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":9714314,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":2576,"num_write":25,"num_write_kb":6863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"20'10","reported_seq":39,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.530910+0000","last_change":"2026-02-25T08:41:36.530910+0000","last_active":"2026-02-25T08:41:36.530910+0000","last_peered":"2026-02-25T08:41:36.530910+0000","last_clean":"2026-02-25T08:41:36.530910+0000","last_became_active":"2026-02-25T08:41:31.569266+0000","last_became_peered":"2026-02-25T08:41:31.569266+0000","last_unstale":"2026-02-25T08:41:36.530910+0000","last_undegraded":"2026-02-25T08:41:36.530910+0000","last_fullsized":"2026-02-25T08:41:36.530910+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2873964,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":14,"num_write_kb":3511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"20'3","reported_seq":28,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-25T08:41:35.515874+0000","last_change":"2026-02-25T08:41:31.567513+0000","last_active":"2026-02-25T08:41:35.515874+0000","last_peered":"2026-02-25T08:41:35.515874+0000","last_clean":"2026-02-25T08:41:35.515874+0000","last_became_active":"2026-02-25T08:41:31.559832+0000","last_became_peered":"2026-02-25T08:41:31.559832+0000","last_unstale":"2026-02-25T08:41:35.515874+0000","last_undegraded":"2026-02-25T08:41:35.515874+0000","last_fullsized":"2026-02-25T08:41:35.515874+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151263,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3649,"num_write":4,"num_write_kb":1027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":37,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.523820+0000","last_change":"2026-02-25T08:41:26.373872+0000","last_active":"2026-02-25T08:41:36.523820+0000","last_peered":"2026-02-25T08:41:36.523820+0000","last_clean":"2026-02-25T08:41:36.523820+0000","last_became_active":"2026-02-25T08:41:24.399215+0000","last_became_peered":"2026-02-25T08:41:24.399215+0000","last_unstale":"2026-02-25T08:41:36.523820+0000","last_undegraded":"2026-02-25T08:41:36.523820+0000","last_fullsized":"2026-02-25T08:41:36.523820+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-25T08:41:35.516234+0000","last_change":"2026-02-25T08:41:31.607063+0000","last_active":"2026-02-25T08:41:35.516234+0000","last_peered":"2026-02-25T08:41:35.516234+0000","last_clean":"2026-02-25T08:41:35.516234+0000","last_became_active":"2026-02-25T08:41:31.602407+0000","last_became_peered":"2026-02-25T08:41:31.602407+0000","last_unstale":"2026-02-25T08:41:35.516234+0000","last_undegraded":"2026-02-25T08:41:35.516234+0000","last_fullsized":"2026-02-25T08:41:35.516234+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.526677+0000","last_change":"2026-02-25T08:41:26.374903+0000","last_active":"2026-02-25T08:41:36.526677+0000","last_peered":"2026-02-25T08:41:36.526677+0000","last_clean":"2026-02-25T08:41:36.526677+0000","last_became_active":"2026-02-25T08:41:24.402449+0000","last_became_peered":"2026-02-25T08:41:24.402449+0000","last_unstale":"2026-02-25T08:41:36.526677+0000","last_undegraded":"2026-02-25T08:41:36.526677+0000","last_fullsized":"2026-02-25T08:41:36.526677+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"20'12","reported_seq":46,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.563677+0000","last_change":"2026-02-25T08:41:36.528209+0000","last_active":"2026-02-25T08:41:36.563677+0000","last_peered":"2026-02-25T08:41:36.563677+0000","last_clean":"2026-02-25T08:41:36.563677+0000","last_became_active":"2026-02-25T08:41:31.591938+0000","last_became_peered":"2026-02-25T08:41:31.591938+0000","last_unstale":"2026-02-25T08:41:36.563677+0000","last_undegraded":"2026-02-25T08:41:36.563677+0000","last_fullsized":"2026-02-25T08:41:36.563677+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":4734,"num_write":16,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":39,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.523143+0000","last_change":"2026-02-25T08:41:26.374022+0000","last_active":"2026-02-25T08:41:36.523143+0000","last_peered":"2026-02-25T08:41:36.523143+0000","last_clean":"2026-02-25T08:41:36.523143+0000","last_became_active":"2026-02-25T08:41:24.420798+0000","last_became_peered":"2026-02-25T08:41:24.420798+0000","last_unstale":"2026-02-25T08:41:36.523143+0000","last_undegraded":"2026-02-25T08:41:36.523143+0000","last_fullsized":"2026-02-25T08:41:36.523143+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"20'9","reported_seq":40,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.523892+0000","last_change":"2026-02-25T08:41:36.530070+0000","last_active":"2026-02-25T08:41:36.523892+0000","last_peered":"2026-02-25T08:41:36.523892+0000","last_clean":"2026-02-25T08:41:36.523892+0000","last_became_active":"2026-02-25T08:41:31.623233+0000","last_became_peered":"2026-02-25T08:41:31.623233+0000","last_unstale":"2026-02-25T08:41:36.523892+0000","last_undegraded":"2026-02-25T08:41:36.523892+0000","last_fullsized":"2026-02-25T08:41:36.523892+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840350,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":3971,"num_write":12,"num_write_kb":2773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":35,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-25T08:41:35.514552+0000","last_change":"2026-02-25T08:41:26.371067+0000","last_active":"2026-02-25T08:41:35.514552+0000","last_peered":"2026-02-25T08:41:35.514552+0000","last_clean":"2026-02-25T08:41:35.514552+0000","last_became_active":"2026-02-25T08:41:24.403313+0000","last_became_peered":"2026-02-25T08:41:24.403313+0000","last_unstale":"2026-02-25T08:41:35.514552+0000","last_undegraded":"2026-02-25T08:41:35.514552+0000","last_fullsized":"2026-02-25T08:41:35.514552+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":103,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.521329+0000","last_change":"2026-02-25T08:41:24.427616+0000","last_active":"2026-02-25T08:41:36.521329+0000","last_peered":"2026-02-25T08:41:36.521329+0000","last_clean":"2026-02-25T08:41:36.521329+0000","last_became_active":"2026-02-25T08:41:24.406166+0000","last_became_peered":"2026-02-25T08:41:24.406166+0000","last_unstale":"2026-02-25T08:41:36.521329+0000","last_undegraded":"2026-02-25T08:41:36.521329+0000","last_fullsized":"2026-02-25T08:41:36.521329+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"22'15","reported_seq":50,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.568948+0000","last_change":"2026-02-25T08:41:36.527277+0000","last_active":"2026-02-25T08:41:36.568948+0000","last_peered":"2026-02-25T08:41:36.568948+0000","last_clean":"2026-02-25T08:41:36.568948+0000","last_became_active":"2026-02-25T08:41:31.558932+0000","last_became_peered":"2026-02-25T08:41:31.558932+0000","last_unstale":"2026-02-25T08:41:36.568948+0000","last_undegraded":"2026-02-25T08:41:36.568948+0000","last_fullsized":"2026-02-25T08:41:36.568948+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10201717,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":6448,"num_write":20,"num_write_kb":5371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"20'21","reported_seq":55,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.544377+0000","last_change":"2026-02-25T08:41:36.527923+0000","last_active":"2026-02-25T08:41:36.544377+0000","last_peered":"2026-02-25T08:41:36.544377+0000","last_clean":"2026-02-25T08:41:36.544377+0000","last_became_active":"2026-02-25T08:41:31.602591+0000","last_became_peered":"2026-02-25T08:41:31.602591+0000","last_unstale":"2026-02-25T08:41:36.544377+0000","last_undegraded":"2026-02-25T08:41:36.544377+0000","last_fullsized":"2026-02-25T08:41:36.544377+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17210137,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":4013,"num_write":28,"num_write_kb":7953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-25T08:41:35.516440+0000","last_change":"2026-02-25T08:41:26.371743+0000","last_active":"2026-02-25T08:41:35.516440+0000","last_peered":"2026-02-25T08:41:35.516440+0000","last_clean":"2026-02-25T08:41:35.516440+0000","last_became_active":"2026-02-25T08:41:24.408398+0000","last_became_peered":"2026-02-25T08:41:24.408398+0000","last_unstale":"2026-02-25T08:41:35.516440+0000","last_undegraded":"2026-02-25T08:41:35.516440+0000","last_fullsized":"2026-02-25T08:41:35.516440+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'10","reported_seq":38,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.545722+0000","last_change":"2026-02-25T08:41:36.536184+0000","last_active":"2026-02-25T08:41:36.545722+0000","last_peered":"2026-02-25T08:41:36.545722+0000","last_clean":"2026-02-25T08:41:36.545722+0000","last_became_active":"2026-02-25T08:41:31.559738+0000","last_became_peered":"2026-02-25T08:41:31.559738+0000","last_unstale":"2026-02-25T08:41:36.545722+0000","last_undegraded":"2026-02-25T08:41:36.545722+0000","last_fullsized":"2026-02-25T08:41:36.545722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5126069,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":3214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.524066+0000","last_change":"2026-02-25T08:41:26.375854+0000","last_active":"2026-02-25T08:41:36.524066+0000","last_peered":"2026-02-25T08:41:36.524066+0000","last_clean":"2026-02-25T08:41:36.524066+0000","last_became_active":"2026-02-25T08:41:24.420122+0000","last_became_peered":"2026-02-25T08:41:24.420122+0000","last_unstale":"2026-02-25T08:41:36.524066+0000","last_undegraded":"2026-02-25T08:41:36.524066+0000","last_fullsized":"2026-02-25T08:41:36.524066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"22'24","reported_seq":55,"reported_epoch":22,"state":"active+clean+snaptrim","last_fresh":"2026-02-25T08:41:36.541087+0000","last_change":"2026-02-25T08:41:36.521409+0000","last_active":"2026-02-25T08:41:36.541087+0000","last_peered":"2026-02-25T08:41:36.541087+0000","last_clean":"2026-02-25T08:41:36.541087+0000","last_became_active":"2026-02-25T08:41:31.548939+0000","last_became_peered":"2026-02-25T08:41:31.548939+0000","last_unstale":"2026-02-25T08:41:36.541087+0000","last_undegraded":"2026-02-25T08:41:36.541087+0000","last_fullsized":"2026-02-25T08:41:36.541087+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":12705721,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":4661,"num_write":28,"num_write_kb":8301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.524085+0000","last_change":"2026-02-25T08:41:26.377253+0000","last_active":"2026-02-25T08:41:36.524085+0000","last_peered":"2026-02-25T08:41:36.524085+0000","last_clean":"2026-02-25T08:41:36.524085+0000","last_became_active":"2026-02-25T08:41:24.409565+0000","last_became_peered":"2026-02-25T08:41:24.409565+0000","last_unstale":"2026-02-25T08:41:36.524085+0000","last_undegraded":"2026-02-25T08:41:36.524085+0000","last_fullsized":"2026-02-25T08:41:36.524085+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"21'20","reported_seq":57,"reported_epoch":22,"state":"active+clean+snaptrim","last_fresh":"2026-02-25T08:41:36.523705+0000","last_change":"2026-02-25T08:41:36.523552+0000","last_active":"2026-02-25T08:41:36.523705+0000","last_peered":"2026-02-25T08:41:36.523705+0000","last_clean":"2026-02-25T08:41:36.523705+0000","last_became_active":"2026-02-25T08:41:31.586165+0000","last_became_peered":"2026-02-25T08:41:31.586165+0000","last_unstale":"2026-02-25T08:41:36.523705+0000","last_undegraded":"2026-02-25T08:41:36.523705+0000","last_fullsized":"2026-02-25T08:41:36.523705+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":11913304,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":11138,"num_write":25,"num_write_kb":6537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":37,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.524823+0000","last_change":"2026-02-25T08:41:26.377529+0000","last_active":"2026-02-25T08:41:36.524823+0000","last_peered":"2026-02-25T08:41:36.524823+0000","last_clean":"2026-02-25T08:41:36.524823+0000","last_became_active":"2026-02-25T08:41:24.426559+0000","last_became_peered":"2026-02-25T08:41:24.426559+0000","last_unstale":"2026-02-25T08:41:36.524823+0000","last_undegraded":"2026-02-25T08:41:36.524823+0000","last_fullsized":"2026-02-25T08:41:36.524823+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":114921815,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":168,"num_read_kb":55094,"num_write":234,"num_write_kb":63236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214748299264,"available":128643067904,"internally_reserved":0,"allocated":86105231360,"data_stored":86105231360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32151072768,"internally_reserved":0,"allocated":21536002048,"data_stored":21536002048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96473759744,"internally_reserved":0,"allocated":64587464704,"data_stored":64587464704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640270,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21033416,"kb_used_data":21033416,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31395368,"statfs":{"total":53687074816,"available":32148856832,"internally_reserved":0,"allocated":21538217984,"data_stored":21538217984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640271,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21028484,"kb_used_data":21028484,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31400300,"statfs":{"total":53687074816,"available":32153907200,"internally_reserved":0,"allocated":21533167616,"data_stored":21533167616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21013824,"kb_used_data":21013824,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31414960,"statfs":{"total":53687074816,"available":32168919040,"internally_reserved":0,"allocated":21518155776,"data_stored":21518155776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640271,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21013444,"kb_used_data":21013444,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31415340,"statfs":{"total":53687074816,"available":32169308160,"internally_reserved":0,"allocated":21517766656,"data_stored":21517766656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32168779776,"internally_reserved":0,"allocated":21518295040,"data_stored":21518295040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32153907200,"internally_reserved":0,"allocated":21533167616,"data_stored":21533167616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32151072768,"internally_reserved":0,"allocated":21536002048,"data_stored":21536002048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32151072768,"internally_reserved":0,"allocated":21536002048,"data_stored":21536002048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53687074816,"available":32169308160,"internally_reserved":0,"allocated":21517766656,"data_stored":21517766656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53687074816,"available":32168779776,"internally_reserved":0,"allocated":21518295040,"data_stored":21518295040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53687074816,"available":32153907200,"internally_reserved":0,"allocated":21533167616,"data_stored":21533167616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53687074816,"available":32151072768,"internally_reserved":0,"allocated":21536002048,"data_stored":21536002048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:37.689 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2026-02-25T08:41:37.689 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-25T08:41:37.689 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:37.801 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:37.797+0000 7f1f4bfc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:37.803 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:37.802+0000 7f1f4bfc1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:37.914 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:37.914 INFO:teuthology.orchestra.run.trial169.stderr:dumped all 2026-02-25T08:41:37.922 INFO:teuthology.orchestra.run.trial169.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2026-02-25T08:41:37.511459+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":115512202,"num_objects":52,"num_object_clones":0,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":221,"num_read_kb":55140,"num_write":315,"num_write_kb":64344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":236,"ondisk_log_size":236,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":84089168,"kb_used_data":84089168,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125625968,"statfs":{"total":214748299264,"available":128640991232,"internally_reserved":0,"allocated":86107308032,"data_stored":86107308032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":114921815,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":168,"num_read_kb":55094,"num_write":234,"num_write_kb":63236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011624"},"pg_stats":[{"pgid":"3.f","version":"20'9","reported_seq":45,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.538422+0000","last_change":"2026-02-25T08:41:36.528428+0000","last_active":"2026-02-25T08:41:36.538422+0000","last_peered":"2026-02-25T08:41:36.538422+0000","last_clean":"2026-02-25T08:41:36.538422+0000","last_became_active":"2026-02-25T08:41:31.577339+0000","last_became_peered":"2026-02-25T08:41:31.577339+0000","last_unstale":"2026-02-25T08:41:36.538422+0000","last_undegraded":"2026-02-25T08:41:36.538422+0000","last_fullsized":"2026-02-25T08:41:36.538422+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6386561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":7817,"num_write":12,"num_write_kb":3115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.522682+0000","last_change":"2026-02-25T08:41:36.528396+0000","last_active":"2026-02-25T08:41:36.522682+0000","last_peered":"2026-02-25T08:41:36.522682+0000","last_clean":"2026-02-25T08:41:36.522682+0000","last_became_active":"2026-02-25T08:41:31.567579+0000","last_became_peered":"2026-02-25T08:41:31.567579+0000","last_unstale":"2026-02-25T08:41:36.522682+0000","last_undegraded":"2026-02-25T08:41:36.522682+0000","last_fullsized":"2026-02-25T08:41:36.522682+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'3","reported_seq":32,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.522858+0000","last_change":"2026-02-25T08:41:36.527143+0000","last_active":"2026-02-25T08:41:36.522858+0000","last_peered":"2026-02-25T08:41:36.522858+0000","last_clean":"2026-02-25T08:41:36.522858+0000","last_became_active":"2026-02-25T08:41:31.550562+0000","last_became_peered":"2026-02-25T08:41:31.550562+0000","last_unstale":"2026-02-25T08:41:36.522858+0000","last_undegraded":"2026-02-25T08:41:36.522858+0000","last_fullsized":"2026-02-25T08:41:36.522858+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2789929,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":6073,"num_write":4,"num_write_kb":1343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"20'16","reported_seq":46,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.524807+0000","last_change":"2026-02-25T08:41:36.533516+0000","last_active":"2026-02-25T08:41:36.524807+0000","last_peered":"2026-02-25T08:41:36.524807+0000","last_clean":"2026-02-25T08:41:36.524807+0000","last_became_active":"2026-02-25T08:41:31.571602+0000","last_became_peered":"2026-02-25T08:41:31.571602+0000","last_unstale":"2026-02-25T08:41:36.524807+0000","last_undegraded":"2026-02-25T08:41:36.524807+0000","last_fullsized":"2026-02-25T08:41:36.524807+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9966419,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":5,"num_write":21,"num_write_kb":5481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"20'9","reported_seq":33,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-25T08:41:35.515751+0000","last_change":"2026-02-25T08:41:31.551969+0000","last_active":"2026-02-25T08:41:35.515751+0000","last_peered":"2026-02-25T08:41:35.515751+0000","last_clean":"2026-02-25T08:41:35.515751+0000","last_became_active":"2026-02-25T08:41:31.546602+0000","last_became_peered":"2026-02-25T08:41:31.546602+0000","last_unstale":"2026-02-25T08:41:35.515751+0000","last_undegraded":"2026-02-25T08:41:35.515751+0000","last_fullsized":"2026-02-25T08:41:35.515751+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7226911,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3201,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"21'20","reported_seq":52,"reported_epoch":22,"state":"active+clean+snaptrim","last_fresh":"2026-02-25T08:41:36.525531+0000","last_change":"2026-02-25T08:41:36.525384+0000","last_active":"2026-02-25T08:41:36.525531+0000","last_peered":"2026-02-25T08:41:36.525531+0000","last_clean":"2026-02-25T08:41:36.525531+0000","last_became_active":"2026-02-25T08:41:31.576273+0000","last_became_peered":"2026-02-25T08:41:31.576273+0000","last_unstale":"2026-02-25T08:41:36.525531+0000","last_undegraded":"2026-02-25T08:41:36.525531+0000","last_fullsized":"2026-02-25T08:41:36.525531+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":9714314,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":2576,"num_write":25,"num_write_kb":6863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"20'10","reported_seq":39,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.530910+0000","last_change":"2026-02-25T08:41:36.530910+0000","last_active":"2026-02-25T08:41:36.530910+0000","last_peered":"2026-02-25T08:41:36.530910+0000","last_clean":"2026-02-25T08:41:36.530910+0000","last_became_active":"2026-02-25T08:41:31.569266+0000","last_became_peered":"2026-02-25T08:41:31.569266+0000","last_unstale":"2026-02-25T08:41:36.530910+0000","last_undegraded":"2026-02-25T08:41:36.530910+0000","last_fullsized":"2026-02-25T08:41:36.530910+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2873964,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":14,"num_write_kb":3511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"20'3","reported_seq":28,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-25T08:41:35.515874+0000","last_change":"2026-02-25T08:41:31.567513+0000","last_active":"2026-02-25T08:41:35.515874+0000","last_peered":"2026-02-25T08:41:35.515874+0000","last_clean":"2026-02-25T08:41:35.515874+0000","last_became_active":"2026-02-25T08:41:31.559832+0000","last_became_peered":"2026-02-25T08:41:31.559832+0000","last_unstale":"2026-02-25T08:41:35.515874+0000","last_undegraded":"2026-02-25T08:41:35.515874+0000","last_fullsized":"2026-02-25T08:41:35.515874+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151263,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3649,"num_write":4,"num_write_kb":1027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":37,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.523820+0000","last_change":"2026-02-25T08:41:26.373872+0000","last_active":"2026-02-25T08:41:36.523820+0000","last_peered":"2026-02-25T08:41:36.523820+0000","last_clean":"2026-02-25T08:41:36.523820+0000","last_became_active":"2026-02-25T08:41:24.399215+0000","last_became_peered":"2026-02-25T08:41:24.399215+0000","last_unstale":"2026-02-25T08:41:36.523820+0000","last_undegraded":"2026-02-25T08:41:36.523820+0000","last_fullsized":"2026-02-25T08:41:36.523820+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-25T08:41:35.516234+0000","last_change":"2026-02-25T08:41:31.607063+0000","last_active":"2026-02-25T08:41:35.516234+0000","last_peered":"2026-02-25T08:41:35.516234+0000","last_clean":"2026-02-25T08:41:35.516234+0000","last_became_active":"2026-02-25T08:41:31.602407+0000","last_became_peered":"2026-02-25T08:41:31.602407+0000","last_unstale":"2026-02-25T08:41:35.516234+0000","last_undegraded":"2026-02-25T08:41:35.516234+0000","last_fullsized":"2026-02-25T08:41:35.516234+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":37,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.526677+0000","last_change":"2026-02-25T08:41:26.374903+0000","last_active":"2026-02-25T08:41:36.526677+0000","last_peered":"2026-02-25T08:41:36.526677+0000","last_clean":"2026-02-25T08:41:36.526677+0000","last_became_active":"2026-02-25T08:41:24.402449+0000","last_became_peered":"2026-02-25T08:41:24.402449+0000","last_unstale":"2026-02-25T08:41:36.526677+0000","last_undegraded":"2026-02-25T08:41:36.526677+0000","last_fullsized":"2026-02-25T08:41:36.526677+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"20'12","reported_seq":46,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.563677+0000","last_change":"2026-02-25T08:41:36.528209+0000","last_active":"2026-02-25T08:41:36.563677+0000","last_peered":"2026-02-25T08:41:36.563677+0000","last_clean":"2026-02-25T08:41:36.563677+0000","last_became_active":"2026-02-25T08:41:31.591938+0000","last_became_peered":"2026-02-25T08:41:31.591938+0000","last_unstale":"2026-02-25T08:41:36.563677+0000","last_undegraded":"2026-02-25T08:41:36.563677+0000","last_fullsized":"2026-02-25T08:41:36.563677+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":4734,"num_write":16,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":39,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.523143+0000","last_change":"2026-02-25T08:41:26.374022+0000","last_active":"2026-02-25T08:41:36.523143+0000","last_peered":"2026-02-25T08:41:36.523143+0000","last_clean":"2026-02-25T08:41:36.523143+0000","last_became_active":"2026-02-25T08:41:24.420798+0000","last_became_peered":"2026-02-25T08:41:24.420798+0000","last_unstale":"2026-02-25T08:41:36.523143+0000","last_undegraded":"2026-02-25T08:41:36.523143+0000","last_fullsized":"2026-02-25T08:41:36.523143+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"20'9","reported_seq":40,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.523892+0000","last_change":"2026-02-25T08:41:36.530070+0000","last_active":"2026-02-25T08:41:36.523892+0000","last_peered":"2026-02-25T08:41:36.523892+0000","last_clean":"2026-02-25T08:41:36.523892+0000","last_became_active":"2026-02-25T08:41:31.623233+0000","last_became_peered":"2026-02-25T08:41:31.623233+0000","last_unstale":"2026-02-25T08:41:36.523892+0000","last_undegraded":"2026-02-25T08:41:36.523892+0000","last_fullsized":"2026-02-25T08:41:36.523892+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840350,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":3971,"num_write":12,"num_write_kb":2773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":35,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-25T08:41:35.514552+0000","last_change":"2026-02-25T08:41:26.371067+0000","last_active":"2026-02-25T08:41:35.514552+0000","last_peered":"2026-02-25T08:41:35.514552+0000","last_clean":"2026-02-25T08:41:35.514552+0000","last_became_active":"2026-02-25T08:41:24.403313+0000","last_became_peered":"2026-02-25T08:41:24.403313+0000","last_unstale":"2026-02-25T08:41:35.514552+0000","last_undegraded":"2026-02-25T08:41:35.514552+0000","last_fullsized":"2026-02-25T08:41:35.514552+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":103,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.521329+0000","last_change":"2026-02-25T08:41:24.427616+0000","last_active":"2026-02-25T08:41:36.521329+0000","last_peered":"2026-02-25T08:41:36.521329+0000","last_clean":"2026-02-25T08:41:36.521329+0000","last_became_active":"2026-02-25T08:41:24.406166+0000","last_became_peered":"2026-02-25T08:41:24.406166+0000","last_unstale":"2026-02-25T08:41:36.521329+0000","last_undegraded":"2026-02-25T08:41:36.521329+0000","last_fullsized":"2026-02-25T08:41:36.521329+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"22'15","reported_seq":50,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.568948+0000","last_change":"2026-02-25T08:41:36.527277+0000","last_active":"2026-02-25T08:41:36.568948+0000","last_peered":"2026-02-25T08:41:36.568948+0000","last_clean":"2026-02-25T08:41:36.568948+0000","last_became_active":"2026-02-25T08:41:31.558932+0000","last_became_peered":"2026-02-25T08:41:31.558932+0000","last_unstale":"2026-02-25T08:41:36.568948+0000","last_undegraded":"2026-02-25T08:41:36.568948+0000","last_fullsized":"2026-02-25T08:41:36.568948+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10201717,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":6448,"num_write":20,"num_write_kb":5371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"20'21","reported_seq":55,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.544377+0000","last_change":"2026-02-25T08:41:36.527923+0000","last_active":"2026-02-25T08:41:36.544377+0000","last_peered":"2026-02-25T08:41:36.544377+0000","last_clean":"2026-02-25T08:41:36.544377+0000","last_became_active":"2026-02-25T08:41:31.602591+0000","last_became_peered":"2026-02-25T08:41:31.602591+0000","last_unstale":"2026-02-25T08:41:36.544377+0000","last_undegraded":"2026-02-25T08:41:36.544377+0000","last_fullsized":"2026-02-25T08:41:36.544377+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17210137,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":16,"num_read_kb":4013,"num_write":28,"num_write_kb":7953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":34,"reported_epoch":21,"state":"active+clean","last_fresh":"2026-02-25T08:41:35.516440+0000","last_change":"2026-02-25T08:41:26.371743+0000","last_active":"2026-02-25T08:41:35.516440+0000","last_peered":"2026-02-25T08:41:35.516440+0000","last_clean":"2026-02-25T08:41:35.516440+0000","last_became_active":"2026-02-25T08:41:24.408398+0000","last_became_peered":"2026-02-25T08:41:24.408398+0000","last_unstale":"2026-02-25T08:41:35.516440+0000","last_undegraded":"2026-02-25T08:41:35.516440+0000","last_fullsized":"2026-02-25T08:41:35.516440+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'10","reported_seq":38,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.545722+0000","last_change":"2026-02-25T08:41:36.536184+0000","last_active":"2026-02-25T08:41:36.545722+0000","last_peered":"2026-02-25T08:41:36.545722+0000","last_clean":"2026-02-25T08:41:36.545722+0000","last_became_active":"2026-02-25T08:41:31.559738+0000","last_became_peered":"2026-02-25T08:41:31.559738+0000","last_unstale":"2026-02-25T08:41:36.545722+0000","last_undegraded":"2026-02-25T08:41:36.545722+0000","last_fullsized":"2026-02-25T08:41:36.545722+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5126069,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":3214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":37,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.524066+0000","last_change":"2026-02-25T08:41:26.375854+0000","last_active":"2026-02-25T08:41:36.524066+0000","last_peered":"2026-02-25T08:41:36.524066+0000","last_clean":"2026-02-25T08:41:36.524066+0000","last_became_active":"2026-02-25T08:41:24.420122+0000","last_became_peered":"2026-02-25T08:41:24.420122+0000","last_unstale":"2026-02-25T08:41:36.524066+0000","last_undegraded":"2026-02-25T08:41:36.524066+0000","last_fullsized":"2026-02-25T08:41:36.524066+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"22'24","reported_seq":55,"reported_epoch":22,"state":"active+clean+snaptrim","last_fresh":"2026-02-25T08:41:36.541087+0000","last_change":"2026-02-25T08:41:36.521409+0000","last_active":"2026-02-25T08:41:36.541087+0000","last_peered":"2026-02-25T08:41:36.541087+0000","last_clean":"2026-02-25T08:41:36.541087+0000","last_became_active":"2026-02-25T08:41:31.548939+0000","last_became_peered":"2026-02-25T08:41:31.548939+0000","last_unstale":"2026-02-25T08:41:36.541087+0000","last_undegraded":"2026-02-25T08:41:36.541087+0000","last_fullsized":"2026-02-25T08:41:36.541087+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":12705721,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":4661,"num_write":28,"num_write_kb":8301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":37,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.524085+0000","last_change":"2026-02-25T08:41:26.377253+0000","last_active":"2026-02-25T08:41:36.524085+0000","last_peered":"2026-02-25T08:41:36.524085+0000","last_clean":"2026-02-25T08:41:36.524085+0000","last_became_active":"2026-02-25T08:41:24.409565+0000","last_became_peered":"2026-02-25T08:41:24.409565+0000","last_unstale":"2026-02-25T08:41:36.524085+0000","last_undegraded":"2026-02-25T08:41:36.524085+0000","last_fullsized":"2026-02-25T08:41:36.524085+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"21'20","reported_seq":57,"reported_epoch":22,"state":"active+clean+snaptrim","last_fresh":"2026-02-25T08:41:36.523705+0000","last_change":"2026-02-25T08:41:36.523552+0000","last_active":"2026-02-25T08:41:36.523705+0000","last_peered":"2026-02-25T08:41:36.523705+0000","last_clean":"2026-02-25T08:41:36.523705+0000","last_became_active":"2026-02-25T08:41:31.586165+0000","last_became_peered":"2026-02-25T08:41:31.586165+0000","last_unstale":"2026-02-25T08:41:36.523705+0000","last_undegraded":"2026-02-25T08:41:36.523705+0000","last_fullsized":"2026-02-25T08:41:36.523705+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":11913304,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":11138,"num_write":25,"num_write_kb":6537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":37,"reported_epoch":22,"state":"active+clean","last_fresh":"2026-02-25T08:41:36.524823+0000","last_change":"2026-02-25T08:41:26.377529+0000","last_active":"2026-02-25T08:41:36.524823+0000","last_peered":"2026-02-25T08:41:36.524823+0000","last_clean":"2026-02-25T08:41:36.524823+0000","last_became_active":"2026-02-25T08:41:24.426559+0000","last_became_peered":"2026-02-25T08:41:24.426559+0000","last_unstale":"2026-02-25T08:41:36.524823+0000","last_undegraded":"2026-02-25T08:41:36.524823+0000","last_fullsized":"2026-02-25T08:41:36.524823+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":114921815,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":168,"num_read_kb":55094,"num_write":234,"num_write_kb":63236,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214748299264,"available":128643067904,"internally_reserved":0,"allocated":86105231360,"data_stored":86105231360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32151072768,"internally_reserved":0,"allocated":21536002048,"data_stored":21536002048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96473759744,"internally_reserved":0,"allocated":64587464704,"data_stored":64587464704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640270,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21033416,"kb_used_data":21033416,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31395368,"statfs":{"total":53687074816,"available":32148856832,"internally_reserved":0,"allocated":21538217984,"data_stored":21538217984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640271,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21028484,"kb_used_data":21028484,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31400300,"statfs":{"total":53687074816,"available":32153907200,"internally_reserved":0,"allocated":21533167616,"data_stored":21533167616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21013824,"kb_used_data":21013824,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31414960,"statfs":{"total":53687074816,"available":32168919040,"internally_reserved":0,"allocated":21518155776,"data_stored":21518155776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640271,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21013444,"kb_used_data":21013444,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31415340,"statfs":{"total":53687074816,"available":32169308160,"internally_reserved":0,"allocated":21517766656,"data_stored":21517766656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32168779776,"internally_reserved":0,"allocated":21518295040,"data_stored":21518295040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32153907200,"internally_reserved":0,"allocated":21533167616,"data_stored":21533167616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32151072768,"internally_reserved":0,"allocated":21536002048,"data_stored":21536002048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32151072768,"internally_reserved":0,"allocated":21536002048,"data_stored":21536002048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53687074816,"available":32169308160,"internally_reserved":0,"allocated":21517766656,"data_stored":21517766656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53687074816,"available":32168779776,"internally_reserved":0,"allocated":21518295040,"data_stored":21518295040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53687074816,"available":32153907200,"internally_reserved":0,"allocated":21533167616,"data_stored":21533167616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53687074816,"available":32151072768,"internally_reserved":0,"allocated":21536002048,"data_stored":21536002048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:38.009 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:41:38.085 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:38.082+0000 7fe555374640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:38.087 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:38.086+0000 7fe555374640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:38.518 INFO:tasks.rados.rados.0.trial169.stdout:84: done (0 left) 2026-02-25T08:41:38.518 INFO:tasks.rados.rados.0.trial169.stdout:85: delete oid 19 current snap is 3 2026-02-25T08:41:38.550 INFO:tasks.rados.rados.0.trial169.stdout:85: done (0 left) 2026-02-25T08:41:38.550 INFO:tasks.rados.rados.0.trial169.stdout:86: read oid 2 snap -1 2026-02-25T08:41:38.550 INFO:tasks.rados.rados.0.trial169.stdout:86: expect (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:41:38.550 INFO:tasks.rados.rados.0.trial169.stdout:87: write (excl) oid 50 current snap is 3 2026-02-25T08:41:38.550 INFO:tasks.rados.rados.0.trial169.stdout:87: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2026-02-25T08:41:38.554 INFO:tasks.rados.rados.0.trial169.stdout:87: writing trial16941936-50 from 658841 to 1130352 tid 1 2026-02-25T08:41:38.559 INFO:tasks.rados.rados.0.trial169.stdout:87: writing trial16941936-50 from 1553542 to 2261880 tid 2 2026-02-25T08:41:38.563 INFO:tasks.rados.rados.0.trial169.stdout:87: writing trial16941936-50 from 2964119 to 3058841 tid 3 2026-02-25T08:41:38.563 INFO:tasks.rados.rados.0.trial169.stdout:88: read oid 6 snap -1 2026-02-25T08:41:38.563 INFO:tasks.rados.rados.0.trial169.stdout:88: expect (ObjNum 57 snap 1 seq_num 57) 2026-02-25T08:41:38.563 INFO:tasks.rados.rados.0.trial169.stdout:89: snap_remove snap 2 2026-02-25T08:41:39.523 INFO:tasks.rados.rados.0.trial169.stdout:89: done (3 left) 2026-02-25T08:41:39.524 INFO:tasks.rados.rados.0.trial169.stdout:90: read oid 8 snap -1 2026-02-25T08:41:39.524 INFO:tasks.rados.rados.0.trial169.stdout:90: expect (ObjNum 7 snap 0 seq_num 7) 2026-02-25T08:41:39.524 INFO:tasks.rados.rados.0.trial169.stdout:91: read oid 14 snap -1 2026-02-25T08:41:39.524 INFO:tasks.rados.rados.0.trial169.stdout:91: expect (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:41:39.524 INFO:tasks.rados.rados.0.trial169.stdout:92: snap_create 2026-02-25T08:41:39.524 INFO:tasks.rados.rados.0.trial169.stdout:86: expect (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:41:39.524 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:39.600 INFO:tasks.rados.rados.0.trial169.stdout:88: expect (ObjNum 57 snap 1 seq_num 57) 2026-02-25T08:41:39.643 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1989870 2026-02-25T08:41:39.692 INFO:tasks.rados.rados.0.trial169.stdout:87: finishing write tid 1 to trial16941936-50 2026-02-25T08:41:39.692 INFO:tasks.rados.rados.0.trial169.stdout:87: finishing write tid 2 to trial16941936-50 2026-02-25T08:41:39.692 INFO:tasks.rados.rados.0.trial169.stdout:87: finishing write tid 3 to trial16941936-50 2026-02-25T08:41:39.693 INFO:tasks.rados.rados.0.trial169.stdout:87: finishing write tid 4 to trial16941936-50 2026-02-25T08:41:39.693 INFO:tasks.rados.rados.0.trial169.stdout:87: finishing write tid 5 to trial16941936-50 2026-02-25T08:41:39.693 INFO:tasks.rados.rados.0.trial169.stdout:87: oid 50 updating version 0 to 26 2026-02-25T08:41:39.693 INFO:tasks.rados.rados.0.trial169.stdout:87: oid 50 version 26 is already newer than 26 2026-02-25T08:41:39.693 INFO:tasks.rados.rados.0.trial169.stdout:87: oid 50 version 26 is already newer than 26 2026-02-25T08:41:39.693 INFO:tasks.rados.rados.0.trial169.stdout:87: oid 50 version 26 is already newer than 23 2026-02-25T08:41:39.693 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 50 v 26 (ObjNum 63 snap 3 seq_num 63) dirty exists 2026-02-25T08:41:39.693 INFO:tasks.rados.rados.0.trial169.stdout:87: left oid 50 (ObjNum 63 snap 3 seq_num 63) 2026-02-25T08:41:39.693 INFO:tasks.rados.rados.0.trial169.stdout:91: expect (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:41:39.761 INFO:tasks.rados.rados.0.trial169.stdout:90: expect (ObjNum 7 snap 0 seq_num 7) 2026-02-25T08:41:39.778 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:39.796 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:40.147 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:41:40.225 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:40.222+0000 7f19e25b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:40.228 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:40.226+0000 7f19e25b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:40.525 INFO:tasks.rados.rados.0.trial169.stdout:86: done (5 left) 2026-02-25T08:41:40.525 INFO:tasks.rados.rados.0.trial169.stdout:87: done (4 left) 2026-02-25T08:41:40.525 INFO:tasks.rados.rados.0.trial169.stdout:88: done (3 left) 2026-02-25T08:41:40.525 INFO:tasks.rados.rados.0.trial169.stdout:90: done (2 left) 2026-02-25T08:41:40.525 INFO:tasks.rados.rados.0.trial169.stdout:91: done (1 left) 2026-02-25T08:41:40.525 INFO:tasks.rados.rados.0.trial169.stdout:92: done (0 left) 2026-02-25T08:41:40.526 INFO:tasks.rados.rados.0.trial169.stdout:93: write oid 42 current snap is 4 2026-02-25T08:41:40.526 INFO:tasks.rados.rados.0.trial169.stdout:93: seq_num 64 ranges {675648=546760,2012024=568349} 2026-02-25T08:41:40.529 INFO:tasks.rados.rados.0.trial169.stdout:93: writing trial16941936-42 from 675648 to 1222408 tid 1 2026-02-25T08:41:40.535 INFO:tasks.rados.rados.0.trial169.stdout:93: writing trial16941936-42 from 2012024 to 2580373 tid 2 2026-02-25T08:41:40.535 INFO:tasks.rados.rados.0.trial169.stdout:94: snap_remove snap 3 2026-02-25T08:41:41.528 INFO:tasks.rados.rados.0.trial169.stdout:94: done (1 left) 2026-02-25T08:41:41.528 INFO:tasks.rados.rados.0.trial169.stdout:95: snap_remove snap 1 2026-02-25T08:41:42.287 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:41:42.364 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:42.361+0000 7fc3c26c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:42.367 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:42.365+0000 7fc3c26c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:42.529 INFO:tasks.rados.rados.0.trial169.stdout:95: done (1 left) 2026-02-25T08:41:42.529 INFO:tasks.rados.rados.0.trial169.stdout:96: write (excl) oid 41 current snap is 4 2026-02-25T08:41:42.529 INFO:tasks.rados.rados.0.trial169.stdout:93: finishing write tid 1 to trial16941936-42 2026-02-25T08:41:42.530 INFO:tasks.rados.rados.0.trial169.stdout:96: seq_num 65 ranges {692455=622009,2070506=712007} 2026-02-25T08:41:42.534 INFO:tasks.rados.rados.0.trial169.stdout:96: writing trial16941936-41 from 692455 to 1314464 tid 1 2026-02-25T08:41:42.540 INFO:tasks.rados.rados.0.trial169.stdout:96: writing trial16941936-41 from 2070506 to 2782513 tid 2 2026-02-25T08:41:42.541 INFO:tasks.rados.rados.0.trial169.stdout:97: write oid 35 current snap is 4 2026-02-25T08:41:42.541 INFO:tasks.rados.rados.0.trial169.stdout:93: finishing write tid 2 to trial16941936-42 2026-02-25T08:41:42.541 INFO:tasks.rados.rados.0.trial169.stdout:93: finishing write tid 3 to trial16941936-42 2026-02-25T08:41:42.541 INFO:tasks.rados.rados.0.trial169.stdout:93: finishing write tid 4 to trial16941936-42 2026-02-25T08:41:42.541 INFO:tasks.rados.rados.0.trial169.stdout:93: oid 42 updating version 0 to 29 2026-02-25T08:41:42.541 INFO:tasks.rados.rados.0.trial169.stdout:93: oid 42 version 29 is already newer than 29 2026-02-25T08:41:42.541 INFO:tasks.rados.rados.0.trial169.stdout:93: oid 42 version 29 is already newer than 29 2026-02-25T08:41:42.541 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 42 v 29 (ObjNum 64 snap 4 seq_num 64) dirty exists 2026-02-25T08:41:42.541 INFO:tasks.rados.rados.0.trial169.stdout:93: left oid 42 (ObjNum 64 snap 4 seq_num 64) 2026-02-25T08:41:42.541 INFO:tasks.rados.rados.0.trial169.stdout:97: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2026-02-25T08:41:42.545 INFO:tasks.rados.rados.0.trial169.stdout:97: writing trial16941936-35 from 709262 to 1406520 tid 1 2026-02-25T08:41:42.550 INFO:tasks.rados.rados.0.trial169.stdout:97: writing trial16941936-35 from 2012635 to 2584653 tid 2 2026-02-25T08:41:42.552 INFO:tasks.rados.rados.0.trial169.stdout:97: writing trial16941936-35 from 3046388 to 3109262 tid 3 2026-02-25T08:41:42.552 INFO:tasks.rados.rados.0.trial169.stdout:93: done (2 left) 2026-02-25T08:41:42.552 INFO:tasks.rados.rados.0.trial169.stdout:98: snap_create 2026-02-25T08:41:42.564 INFO:tasks.rados.rados.0.trial169.stdout:96: finishing write tid 1 to trial16941936-41 2026-02-25T08:41:42.577 INFO:tasks.rados.rados.0.trial169.stdout:96: finishing write tid 2 to trial16941936-41 2026-02-25T08:41:42.586 INFO:tasks.rados.rados.0.trial169.stdout:96: finishing write tid 3 to trial16941936-41 2026-02-25T08:41:42.587 INFO:tasks.rados.rados.0.trial169.stdout:96: finishing write tid 4 to trial16941936-41 2026-02-25T08:41:42.588 INFO:tasks.rados.rados.0.trial169.stdout:96: oid 41 updating version 0 to 25 2026-02-25T08:41:42.588 INFO:tasks.rados.rados.0.trial169.stdout:96: oid 41 version 25 is already newer than 23 2026-02-25T08:41:42.588 INFO:tasks.rados.rados.0.trial169.stdout:96: oid 41 version 25 is already newer than 25 2026-02-25T08:41:42.588 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 41 v 25 (ObjNum 65 snap 4 seq_num 65) dirty exists 2026-02-25T08:41:42.588 INFO:tasks.rados.rados.0.trial169.stdout:96: left oid 41 (ObjNum 65 snap 4 seq_num 65) 2026-02-25T08:41:42.599 INFO:tasks.rados.rados.0.trial169.stdout:97: finishing write tid 1 to trial16941936-35 2026-02-25T08:41:42.604 INFO:tasks.rados.rados.0.trial169.stdout:97: finishing write tid 2 to trial16941936-35 2026-02-25T08:41:42.609 INFO:tasks.rados.rados.0.trial169.stdout:97: finishing write tid 3 to trial16941936-35 2026-02-25T08:41:42.613 INFO:tasks.rados.rados.0.trial169.stdout:97: finishing write tid 4 to trial16941936-35 2026-02-25T08:41:42.615 INFO:tasks.rados.rados.0.trial169.stdout:97: finishing write tid 5 to trial16941936-35 2026-02-25T08:41:42.615 INFO:tasks.rados.rados.0.trial169.stdout:97: oid 35 updating version 0 to 34 2026-02-25T08:41:42.615 INFO:tasks.rados.rados.0.trial169.stdout:97: oid 35 version 34 is already newer than 34 2026-02-25T08:41:42.615 INFO:tasks.rados.rados.0.trial169.stdout:97: oid 35 version 34 is already newer than 34 2026-02-25T08:41:42.615 INFO:tasks.rados.rados.0.trial169.stdout:97: oid 35 version 34 is already newer than 34 2026-02-25T08:41:42.615 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 35 v 34 (ObjNum 66 snap 4 seq_num 66) dirty exists 2026-02-25T08:41:42.615 INFO:tasks.rados.rados.0.trial169.stdout:97: left oid 35 (ObjNum 66 snap 4 seq_num 66) 2026-02-25T08:41:42.925 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-25T08:41:42.925 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2026-02-25T08:41:43.004 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:43.000+0000 7f16ac5d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:43.007 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:43.005+0000 7f16ac5d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:43.533 INFO:teuthology.orchestra.run.trial169.stderr:reweighted osd.0 to 1 (10000) 2026-02-25T08:41:43.535 INFO:tasks.rados.rados.0.trial169.stdout:96: done (2 left) 2026-02-25T08:41:43.535 INFO:tasks.rados.rados.0.trial169.stdout:97: done (1 left) 2026-02-25T08:41:43.535 INFO:tasks.rados.rados.0.trial169.stdout:98: done (0 left) 2026-02-25T08:41:43.535 INFO:tasks.rados.rados.0.trial169.stdout:99: write (excl) oid 42 current snap is 5 2026-02-25T08:41:43.535 INFO:tasks.rados.rados.0.trial169.stdout:99: seq_num 67 ranges {726069=772507,2071117=715676} 2026-02-25T08:41:43.541 INFO:tasks.rados.rados.0.trial169.stdout:99: writing trial16941936-42 from 726069 to 1498576 tid 1 2026-02-25T08:41:43.548 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2026-02-25T08:41:43.548 INFO:tasks.rados.rados.0.trial169.stdout:99: writing trial16941936-42 from 2071117 to 2786793 tid 2 2026-02-25T08:41:43.549 INFO:tasks.rados.rados.0.trial169.stdout:100: snap_remove snap 4 2026-02-25T08:41:43.625 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:43.621+0000 7eff49246640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:43.628 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:43.626+0000 7eff49246640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:44.425 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:41:44.502 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:44.500+0000 7f380559c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:44.504 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:44.503+0000 7f380559c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:44.536 INFO:teuthology.orchestra.run.trial169.stderr:reweighted osd.1 to 1 (10000) 2026-02-25T08:41:44.538 INFO:tasks.rados.rados.0.trial169.stdout:100: done (1 left) 2026-02-25T08:41:44.538 INFO:tasks.rados.rados.0.trial169.stdout:101: read oid 2 snap -1 2026-02-25T08:41:44.538 INFO:tasks.rados.rados.0.trial169.stdout:101: expect (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:41:44.538 INFO:tasks.rados.rados.0.trial169.stdout:102: read oid 49 snap -1 2026-02-25T08:41:44.538 INFO:tasks.rados.rados.0.trial169.stdout:102: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:41:44.538 INFO:tasks.rados.rados.0.trial169.stdout:103: read oid 38 snap -1 2026-02-25T08:41:44.539 INFO:tasks.rados.rados.0.trial169.stdout:103: expect (ObjNum 37 snap 0 seq_num 37) 2026-02-25T08:41:44.539 INFO:tasks.rados.rados.0.trial169.stdout:104: write oid 17 current snap is 5 2026-02-25T08:41:44.539 INFO:tasks.rados.rados.0.trial169.stdout:99: finishing write tid 1 to trial16941936-42 2026-02-25T08:41:44.539 INFO:tasks.rados.rados.0.trial169.stdout:104: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2026-02-25T08:41:44.542 INFO:tasks.rados.rados.0.trial169.stdout:104: writing trial16941936-17 from 742876 to 1190632 tid 1 2026-02-25T08:41:44.547 INFO:tasks.rados.rados.0.trial169.stdout:104: writing trial16941936-17 from 1729599 to 2305286 tid 2 2026-02-25T08:41:44.548 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2026-02-25T08:41:44.552 INFO:tasks.rados.rados.0.trial169.stdout:104: writing trial16941936-17 from 3101234 to 3142876 tid 3 2026-02-25T08:41:44.552 INFO:tasks.rados.rados.0.trial169.stdout:105: read oid 48 snap -1 2026-02-25T08:41:44.552 INFO:tasks.rados.rados.0.trial169.stdout:105: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:44.552 INFO:tasks.rados.rados.0.trial169.stdout:106: delete oid 32 current snap is 5 2026-02-25T08:41:44.552 INFO:tasks.rados.rados.0.trial169.stdout:99: finishing write tid 2 to trial16941936-42 2026-02-25T08:41:44.552 INFO:tasks.rados.rados.0.trial169.stdout:99: finishing write tid 3 to trial16941936-42 2026-02-25T08:41:44.552 INFO:tasks.rados.rados.0.trial169.stdout:99: finishing write tid 4 to trial16941936-42 2026-02-25T08:41:44.552 INFO:tasks.rados.rados.0.trial169.stdout:99: oid 42 updating version 0 to 33 2026-02-25T08:41:44.552 INFO:tasks.rados.rados.0.trial169.stdout:99: oid 42 updating version 33 to 35 2026-02-25T08:41:44.552 INFO:tasks.rados.rados.0.trial169.stdout:99: oid 42 version 35 is already newer than 35 2026-02-25T08:41:44.553 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 42 v 35 (ObjNum 67 snap 5 seq_num 67) dirty exists 2026-02-25T08:41:44.553 INFO:tasks.rados.rados.0.trial169.stdout:99: left oid 42 (ObjNum 67 snap 5 seq_num 67) 2026-02-25T08:41:44.563 INFO:tasks.rados.rados.0.trial169.stdout:102: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:41:44.628 INFO:tasks.rados.rados.0.trial169.stdout:99: done (6 left) 2026-02-25T08:41:44.628 INFO:tasks.rados.rados.0.trial169.stdout:102: done (5 left) 2026-02-25T08:41:44.628 INFO:tasks.rados.rados.0.trial169.stdout:106: done (4 left) 2026-02-25T08:41:44.628 INFO:tasks.rados.rados.0.trial169.stdout:107: write (excl) oid 28 current snap is 5 2026-02-25T08:41:44.629 INFO:tasks.rados.rados.0.trial169.stdout:103: expect (ObjNum 37 snap 0 seq_num 37) 2026-02-25T08:41:44.630 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:44.625+0000 7f67de6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:44.633 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:44.631+0000 7f67de6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:44.682 INFO:tasks.rados.rados.0.trial169.stdout:101: expect (ObjNum 50 snap 0 seq_num 50) 2026-02-25T08:41:44.682 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:44.720 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1469399 2026-02-25T08:41:44.753 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2475839 2026-02-25T08:41:44.756 INFO:tasks.rados.rados.0.trial169.stdout:104: finishing write tid 1 to trial16941936-17 2026-02-25T08:41:44.756 INFO:tasks.rados.rados.0.trial169.stdout:107: seq_num 70 ranges {776490=714607,2246563=579356} 2026-02-25T08:41:44.762 INFO:tasks.rados.rados.0.trial169.stdout:107: writing trial16941936-28 from 776490 to 1491097 tid 1 2026-02-25T08:41:44.767 INFO:tasks.rados.rados.0.trial169.stdout:107: writing trial16941936-28 from 2246563 to 2825919 tid 2 2026-02-25T08:41:44.768 INFO:tasks.rados.rados.0.trial169.stdout:101: done (4 left) 2026-02-25T08:41:44.768 INFO:tasks.rados.rados.0.trial169.stdout:103: done (3 left) 2026-02-25T08:41:44.768 INFO:tasks.rados.rados.0.trial169.stdout:108: delete oid 38 current snap is 5 2026-02-25T08:41:44.768 INFO:tasks.rados.rados.0.trial169.stdout:105: expect (ObjNum 47 snap 0 seq_num 47) 2026-02-25T08:41:44.770 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1594103 2026-02-25T08:41:44.823 INFO:tasks.rados.rados.0.trial169.stdout:104: finishing write tid 2 to trial16941936-17 2026-02-25T08:41:44.823 INFO:tasks.rados.rados.0.trial169.stdout:104: finishing write tid 3 to trial16941936-17 2026-02-25T08:41:44.823 INFO:tasks.rados.rados.0.trial169.stdout:104: finishing write tid 4 to trial16941936-17 2026-02-25T08:41:44.823 INFO:tasks.rados.rados.0.trial169.stdout:104: finishing write tid 5 to trial16941936-17 2026-02-25T08:41:44.823 INFO:tasks.rados.rados.0.trial169.stdout:104: oid 17 updating version 0 to 20 2026-02-25T08:41:44.823 INFO:tasks.rados.rados.0.trial169.stdout:104: oid 17 version 20 is already newer than 20 2026-02-25T08:41:44.823 INFO:tasks.rados.rados.0.trial169.stdout:104: oid 17 version 20 is already newer than 20 2026-02-25T08:41:44.823 INFO:tasks.rados.rados.0.trial169.stdout:104: oid 17 version 20 is already newer than 20 2026-02-25T08:41:44.823 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 17 v 20 (ObjNum 68 snap 5 seq_num 68) dirty exists 2026-02-25T08:41:44.823 INFO:tasks.rados.rados.0.trial169.stdout:104: left oid 17 (ObjNum 68 snap 5 seq_num 68) 2026-02-25T08:41:44.824 INFO:tasks.rados.rados.0.trial169.stdout:107: finishing write tid 1 to trial16941936-28 2026-02-25T08:41:44.824 INFO:tasks.rados.rados.0.trial169.stdout:107: finishing write tid 2 to trial16941936-28 2026-02-25T08:41:44.824 INFO:tasks.rados.rados.0.trial169.stdout:107: finishing write tid 3 to trial16941936-28 2026-02-25T08:41:44.824 INFO:tasks.rados.rados.0.trial169.stdout:107: finishing write tid 4 to trial16941936-28 2026-02-25T08:41:44.824 INFO:tasks.rados.rados.0.trial169.stdout:107: oid 28 updating version 0 to 12 2026-02-25T08:41:44.824 INFO:tasks.rados.rados.0.trial169.stdout:107: oid 28 updating version 12 to 14 2026-02-25T08:41:44.824 INFO:tasks.rados.rados.0.trial169.stdout:107: oid 28 version 14 is already newer than 14 2026-02-25T08:41:44.824 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 28 v 14 (ObjNum 70 snap 5 seq_num 70) dirty exists 2026-02-25T08:41:44.824 INFO:tasks.rados.rados.0.trial169.stdout:107: left oid 28 (ObjNum 70 snap 5 seq_num 70) 2026-02-25T08:41:44.824 INFO:tasks.rados.rados.0.trial169.stdout:104: done (3 left) 2026-02-25T08:41:44.824 INFO:tasks.rados.rados.0.trial169.stdout:105: done (2 left) 2026-02-25T08:41:44.824 INFO:tasks.rados.rados.0.trial169.stdout:107: done (1 left) 2026-02-25T08:41:44.824 INFO:tasks.rados.rados.0.trial169.stdout:108: done (0 left) 2026-02-25T08:41:44.824 INFO:tasks.rados.rados.0.trial169.stdout:109: write (excl) oid 40 current snap is 5 2026-02-25T08:41:44.825 INFO:tasks.rados.rados.0.trial169.stdout:109: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2026-02-25T08:41:44.826 INFO:tasks.rados.rados.0.trial169.stdout:109: writing trial16941936-40 from 410104 to 875209 tid 1 2026-02-25T08:41:44.830 INFO:tasks.rados.rados.0.trial169.stdout:109: writing trial16941936-40 from 1563527 to 2146552 tid 2 2026-02-25T08:41:44.835 INFO:tasks.rados.rados.0.trial169.stdout:109: writing trial16941936-40 from 2810926 to 3210104 tid 3 2026-02-25T08:41:44.835 INFO:tasks.rados.rados.0.trial169.stdout:110: read oid 26 snap -1 2026-02-25T08:41:44.835 INFO:tasks.rados.rados.0.trial169.stdout:110: expect (ObjNum 25 snap 0 seq_num 25) 2026-02-25T08:41:44.835 INFO:tasks.rados.rados.0.trial169.stdout:111: delete oid 42 current snap is 5 2026-02-25T08:41:44.842 INFO:tasks.rados.rados.0.trial169.stdout:109: finishing write tid 1 to trial16941936-40 2026-02-25T08:41:44.845 INFO:tasks.rados.rados.0.trial169.stdout:110: expect (ObjNum 25 snap 0 seq_num 25) 2026-02-25T08:41:44.906 INFO:tasks.rados.rados.0.trial169.stdout:109: finishing write tid 2 to trial16941936-40 2026-02-25T08:41:44.906 INFO:tasks.rados.rados.0.trial169.stdout:109: finishing write tid 3 to trial16941936-40 2026-02-25T08:41:44.906 INFO:tasks.rados.rados.0.trial169.stdout:110: done (2 left) 2026-02-25T08:41:44.907 INFO:tasks.rados.rados.0.trial169.stdout:111: done (1 left) 2026-02-25T08:41:44.907 INFO:tasks.rados.rados.0.trial169.stdout:112: snap_create 2026-02-25T08:41:44.907 INFO:tasks.rados.rados.0.trial169.stdout:109: finishing write tid 4 to trial16941936-40 2026-02-25T08:41:44.907 INFO:tasks.rados.rados.0.trial169.stdout:109: finishing write tid 5 to trial16941936-40 2026-02-25T08:41:44.907 INFO:tasks.rados.rados.0.trial169.stdout:109: oid 40 updating version 0 to 38 2026-02-25T08:41:44.907 INFO:tasks.rados.rados.0.trial169.stdout:109: oid 40 version 38 is already newer than 38 2026-02-25T08:41:44.907 INFO:tasks.rados.rados.0.trial169.stdout:109: oid 40 version 38 is already newer than 35 2026-02-25T08:41:44.907 INFO:tasks.rados.rados.0.trial169.stdout:109: oid 40 version 38 is already newer than 38 2026-02-25T08:41:44.907 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 40 v 38 (ObjNum 72 snap 5 seq_num 72) dirty exists 2026-02-25T08:41:44.907 INFO:tasks.rados.rados.0.trial169.stdout:109: left oid 40 (ObjNum 72 snap 5 seq_num 72) 2026-02-25T08:41:45.537 INFO:teuthology.orchestra.run.trial169.stderr:reweighted osd.2 to 1 (10000) 2026-02-25T08:41:45.540 INFO:tasks.rados.rados.0.trial169.stdout:109: done (1 left) 2026-02-25T08:41:45.541 INFO:tasks.rados.rados.0.trial169.stdout:112: done (0 left) 2026-02-25T08:41:45.541 INFO:tasks.rados.rados.0.trial169.stdout:113: delete oid 48 current snap is 6 2026-02-25T08:41:45.554 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2026-02-25T08:41:45.572 INFO:tasks.rados.rados.0.trial169.stdout:113: done (0 left) 2026-02-25T08:41:45.573 INFO:tasks.rados.rados.0.trial169.stdout:114: read oid 45 snap 5 2026-02-25T08:41:45.573 INFO:tasks.rados.rados.0.trial169.stdout:114: expect (ObjNum 53 snap 1 seq_num 53) 2026-02-25T08:41:45.573 INFO:tasks.rados.rados.0.trial169.stdout:115: snap_create 2026-02-25T08:41:45.595 INFO:tasks.rados.rados.0.trial169.stdout:114: expect (ObjNum 53 snap 1 seq_num 53) 2026-02-25T08:41:45.633 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:45.628+0000 7f267e1a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:45.635 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:45.633+0000 7f267e1a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:45.669 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1632643 2026-02-25T08:41:46.543 INFO:teuthology.orchestra.run.trial169.stderr:reweighted osd.3 to 1 (10000) 2026-02-25T08:41:46.547 INFO:tasks.rados.rados.0.trial169.stdout:114: done (1 left) 2026-02-25T08:41:46.547 INFO:tasks.rados.rados.0.trial169.stdout:115: done (0 left) 2026-02-25T08:41:46.547 INFO:tasks.rados.rados.0.trial169.stdout:116: snap_create 2026-02-25T08:41:46.558 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2026-02-25T08:41:46.558 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:46.580 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:41:46.674 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:46.671+0000 7ff0932cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:46.676 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:46.672+0000 7f4a76acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:46.676 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:46.675+0000 7ff0932cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:46.678 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:46.677+0000 7f4a76acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:46.790 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:46.790 INFO:teuthology.orchestra.run.trial169.stderr:dumped all 2026-02-25T08:41:46.799 INFO:teuthology.orchestra.run.trial169.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2026-02-25T08:41:46.542621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112638271,"num_objects":56,"num_object_clones":4,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":255,"num_read_kb":73726,"num_write":332,"num_write_kb":68091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":263,"ondisk_log_size":263,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":84093864,"kb_used_data":84093864,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125621272,"statfs":{"total":214748299264,"available":128636182528,"internally_reserved":0,"allocated":86112116736,"data_stored":86112116736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015916"},"pg_stats":[{"pgid":"3.f","version":"23'11","reported_seq":63,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.546810+0000","last_change":"2026-02-25T08:41:41.550669+0000","last_active":"2026-02-25T08:41:41.546810+0000","last_peered":"2026-02-25T08:41:41.546810+0000","last_clean":"2026-02-25T08:41:41.546810+0000","last_became_active":"2026-02-25T08:41:31.577339+0000","last_became_peered":"2026-02-25T08:41:31.577339+0000","last_unstale":"2026-02-25T08:41:41.546810+0000","last_undegraded":"2026-02-25T08:41:41.546810+0000","last_fullsized":"2026-02-25T08:41:41.546810+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4319333,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":18,"num_read_kb":7817,"num_write":13,"num_write_kb":3115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":42,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.544890+0000","last_change":"2026-02-25T08:41:41.552033+0000","last_active":"2026-02-25T08:41:41.544890+0000","last_peered":"2026-02-25T08:41:41.544890+0000","last_clean":"2026-02-25T08:41:41.544890+0000","last_became_active":"2026-02-25T08:41:31.567579+0000","last_became_peered":"2026-02-25T08:41:31.567579+0000","last_unstale":"2026-02-25T08:41:41.544890+0000","last_undegraded":"2026-02-25T08:41:41.544890+0000","last_fullsized":"2026-02-25T08:41:41.544890+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'3","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.538452+0000","last_change":"2026-02-25T08:41:41.545268+0000","last_active":"2026-02-25T08:41:41.538452+0000","last_peered":"2026-02-25T08:41:41.538452+0000","last_clean":"2026-02-25T08:41:41.538452+0000","last_became_active":"2026-02-25T08:41:31.550562+0000","last_became_peered":"2026-02-25T08:41:31.550562+0000","last_unstale":"2026-02-25T08:41:41.538452+0000","last_undegraded":"2026-02-25T08:41:41.538452+0000","last_fullsized":"2026-02-25T08:41:41.538452+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2789929,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":9709,"num_write":4,"num_write_kb":1343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"20'16","reported_seq":66,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.543412+0000","last_change":"2026-02-25T08:41:41.549102+0000","last_active":"2026-02-25T08:41:41.543412+0000","last_peered":"2026-02-25T08:41:41.543412+0000","last_clean":"2026-02-25T08:41:41.543412+0000","last_became_active":"2026-02-25T08:41:31.571602+0000","last_became_peered":"2026-02-25T08:41:31.571602+0000","last_unstale":"2026-02-25T08:41:41.543412+0000","last_undegraded":"2026-02-25T08:41:41.543412+0000","last_fullsized":"2026-02-25T08:41:41.543412+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9966419,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":3790,"num_write":21,"num_write_kb":5481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"22'13","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.551408+0000","last_change":"2026-02-25T08:41:44.554216+0000","last_active":"2026-02-25T08:41:45.551408+0000","last_peered":"2026-02-25T08:41:45.551408+0000","last_clean":"2026-02-25T08:41:45.551408+0000","last_became_active":"2026-02-25T08:41:31.546602+0000","last_became_peered":"2026-02-25T08:41:31.546602+0000","last_unstale":"2026-02-25T08:41:45.551408+0000","last_undegraded":"2026-02-25T08:41:45.551408+0000","last_fullsized":"2026-02-25T08:41:45.551408+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8100875,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":5693,"num_write":17,"num_write_kb":4612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"22'22","reported_seq":71,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.541238+0000","last_change":"2026-02-25T08:41:41.546892+0000","last_active":"2026-02-25T08:41:41.541238+0000","last_peered":"2026-02-25T08:41:41.541238+0000","last_clean":"2026-02-25T08:41:41.541238+0000","last_became_active":"2026-02-25T08:41:31.576273+0000","last_became_peered":"2026-02-25T08:41:31.576273+0000","last_unstale":"2026-02-25T08:41:41.541238+0000","last_undegraded":"2026-02-25T08:41:41.541238+0000","last_fullsized":"2026-02-25T08:41:41.541238+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":9714314,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":7567,"num_write":25,"num_write_kb":6863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"22'11","reported_seq":56,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.542656+0000","last_change":"2026-02-25T08:41:41.548787+0000","last_active":"2026-02-25T08:41:41.542656+0000","last_peered":"2026-02-25T08:41:41.542656+0000","last_clean":"2026-02-25T08:41:41.542656+0000","last_became_active":"2026-02-25T08:41:31.569266+0000","last_became_peered":"2026-02-25T08:41:31.569266+0000","last_unstale":"2026-02-25T08:41:41.542656+0000","last_undegraded":"2026-02-25T08:41:41.542656+0000","last_fullsized":"2026-02-25T08:41:41.542656+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2873964,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":14,"num_write_kb":3511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"20'3","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.551054+0000","last_change":"2026-02-25T08:41:44.552340+0000","last_active":"2026-02-25T08:41:45.551054+0000","last_peered":"2026-02-25T08:41:45.551054+0000","last_clean":"2026-02-25T08:41:45.551054+0000","last_became_active":"2026-02-25T08:41:31.559832+0000","last_became_peered":"2026-02-25T08:41:31.559832+0000","last_unstale":"2026-02-25T08:41:45.551054+0000","last_undegraded":"2026-02-25T08:41:45.551054+0000","last_fullsized":"2026-02-25T08:41:45.551054+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151263,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3649,"num_write":4,"num_write_kb":1027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.541022+0000","last_change":"2026-02-25T08:41:26.373872+0000","last_active":"2026-02-25T08:41:41.541022+0000","last_peered":"2026-02-25T08:41:41.541022+0000","last_clean":"2026-02-25T08:41:41.541022+0000","last_became_active":"2026-02-25T08:41:24.399215+0000","last_became_peered":"2026-02-25T08:41:24.399215+0000","last_unstale":"2026-02-25T08:41:41.541022+0000","last_undegraded":"2026-02-25T08:41:41.541022+0000","last_fullsized":"2026-02-25T08:41:41.541022+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.551093+0000","last_change":"2026-02-25T08:41:44.553752+0000","last_active":"2026-02-25T08:41:45.551093+0000","last_peered":"2026-02-25T08:41:45.551093+0000","last_clean":"2026-02-25T08:41:45.551093+0000","last_became_active":"2026-02-25T08:41:31.602407+0000","last_became_peered":"2026-02-25T08:41:31.602407+0000","last_unstale":"2026-02-25T08:41:45.551093+0000","last_undegraded":"2026-02-25T08:41:45.551093+0000","last_fullsized":"2026-02-25T08:41:45.551093+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.542652+0000","last_change":"2026-02-25T08:41:26.374903+0000","last_active":"2026-02-25T08:41:41.542652+0000","last_peered":"2026-02-25T08:41:41.542652+0000","last_clean":"2026-02-25T08:41:41.542652+0000","last_became_active":"2026-02-25T08:41:24.402449+0000","last_became_peered":"2026-02-25T08:41:24.402449+0000","last_unstale":"2026-02-25T08:41:41.542652+0000","last_undegraded":"2026-02-25T08:41:41.542652+0000","last_fullsized":"2026-02-25T08:41:41.542652+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"20'12","reported_seq":63,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.546271+0000","last_change":"2026-02-25T08:41:41.548708+0000","last_active":"2026-02-25T08:41:41.546271+0000","last_peered":"2026-02-25T08:41:41.546271+0000","last_clean":"2026-02-25T08:41:41.546271+0000","last_became_active":"2026-02-25T08:41:31.591938+0000","last_became_peered":"2026-02-25T08:41:31.591938+0000","last_unstale":"2026-02-25T08:41:41.546271+0000","last_undegraded":"2026-02-25T08:41:41.546271+0000","last_fullsized":"2026-02-25T08:41:41.546271+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":4734,"num_write":16,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":54,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.544514+0000","last_change":"2026-02-25T08:41:26.374022+0000","last_active":"2026-02-25T08:41:41.544514+0000","last_peered":"2026-02-25T08:41:41.544514+0000","last_clean":"2026-02-25T08:41:41.544514+0000","last_became_active":"2026-02-25T08:41:24.420798+0000","last_became_peered":"2026-02-25T08:41:24.420798+0000","last_unstale":"2026-02-25T08:41:41.544514+0000","last_undegraded":"2026-02-25T08:41:41.544514+0000","last_fullsized":"2026-02-25T08:41:41.544514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"20'9","reported_seq":57,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.542344+0000","last_change":"2026-02-25T08:41:41.546520+0000","last_active":"2026-02-25T08:41:41.542344+0000","last_peered":"2026-02-25T08:41:41.542344+0000","last_clean":"2026-02-25T08:41:41.542344+0000","last_became_active":"2026-02-25T08:41:31.623233+0000","last_became_peered":"2026-02-25T08:41:31.623233+0000","last_unstale":"2026-02-25T08:41:41.542344+0000","last_undegraded":"2026-02-25T08:41:41.542344+0000","last_fullsized":"2026-02-25T08:41:41.542344+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840350,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":3971,"num_write":12,"num_write_kb":2773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.550121+0000","last_change":"2026-02-25T08:41:26.371067+0000","last_active":"2026-02-25T08:41:45.550121+0000","last_peered":"2026-02-25T08:41:45.550121+0000","last_clean":"2026-02-25T08:41:45.550121+0000","last_became_active":"2026-02-25T08:41:24.403313+0000","last_became_peered":"2026-02-25T08:41:24.403313+0000","last_unstale":"2026-02-25T08:41:45.550121+0000","last_undegraded":"2026-02-25T08:41:45.550121+0000","last_fullsized":"2026-02-25T08:41:45.550121+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":118,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.541622+0000","last_change":"2026-02-25T08:41:24.427616+0000","last_active":"2026-02-25T08:41:41.541622+0000","last_peered":"2026-02-25T08:41:41.541622+0000","last_clean":"2026-02-25T08:41:41.541622+0000","last_became_active":"2026-02-25T08:41:24.406166+0000","last_became_peered":"2026-02-25T08:41:24.406166+0000","last_unstale":"2026-02-25T08:41:41.541622+0000","last_undegraded":"2026-02-25T08:41:41.541622+0000","last_fullsized":"2026-02-25T08:41:41.541622+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"25'19","reported_seq":67,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.562383+0000","last_change":"2026-02-25T08:41:41.564290+0000","last_active":"2026-02-25T08:41:41.562383+0000","last_peered":"2026-02-25T08:41:41.562383+0000","last_clean":"2026-02-25T08:41:41.562383+0000","last_became_active":"2026-02-25T08:41:31.558932+0000","last_became_peered":"2026-02-25T08:41:31.558932+0000","last_unstale":"2026-02-25T08:41:41.562383+0000","last_undegraded":"2026-02-25T08:41:41.562383+0000","last_fullsized":"2026-02-25T08:41:41.562383+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7899191,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":6448,"num_write":21,"num_write_kb":5371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"25'28","reported_seq":76,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.559572+0000","last_change":"2026-02-25T08:41:41.561348+0000","last_active":"2026-02-25T08:41:41.559572+0000","last_peered":"2026-02-25T08:41:41.559572+0000","last_clean":"2026-02-25T08:41:41.559572+0000","last_became_active":"2026-02-25T08:41:31.602591+0000","last_became_peered":"2026-02-25T08:41:31.602591+0000","last_unstale":"2026-02-25T08:41:41.559572+0000","last_undegraded":"2026-02-25T08:41:41.559572+0000","last_fullsized":"2026-02-25T08:41:41.559572+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17445435,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":4014,"num_write":33,"num_write_kb":9199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.550932+0000","last_change":"2026-02-25T08:41:26.371743+0000","last_active":"2026-02-25T08:41:45.550932+0000","last_peered":"2026-02-25T08:41:45.550932+0000","last_clean":"2026-02-25T08:41:45.550932+0000","last_became_active":"2026-02-25T08:41:24.408398+0000","last_became_peered":"2026-02-25T08:41:24.408398+0000","last_unstale":"2026-02-25T08:41:45.550932+0000","last_undegraded":"2026-02-25T08:41:45.550932+0000","last_fullsized":"2026-02-25T08:41:45.550932+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'10","reported_seq":55,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.541375+0000","last_change":"2026-02-25T08:41:41.548902+0000","last_active":"2026-02-25T08:41:41.541375+0000","last_peered":"2026-02-25T08:41:41.541375+0000","last_clean":"2026-02-25T08:41:41.541375+0000","last_became_active":"2026-02-25T08:41:31.559738+0000","last_became_peered":"2026-02-25T08:41:31.559738+0000","last_unstale":"2026-02-25T08:41:41.541375+0000","last_undegraded":"2026-02-25T08:41:41.541375+0000","last_fullsized":"2026-02-25T08:41:41.541375+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5126069,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":3214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.548298+0000","last_change":"2026-02-25T08:41:26.375854+0000","last_active":"2026-02-25T08:41:41.548298+0000","last_peered":"2026-02-25T08:41:41.548298+0000","last_clean":"2026-02-25T08:41:41.548298+0000","last_became_active":"2026-02-25T08:41:24.420122+0000","last_became_peered":"2026-02-25T08:41:24.420122+0000","last_unstale":"2026-02-25T08:41:41.548298+0000","last_undegraded":"2026-02-25T08:41:41.548298+0000","last_fullsized":"2026-02-25T08:41:41.548298+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"26'29","reported_seq":78,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.545146+0000","last_change":"2026-02-25T08:41:41.552766+0000","last_active":"2026-02-25T08:41:41.545146+0000","last_peered":"2026-02-25T08:41:41.545146+0000","last_clean":"2026-02-25T08:41:41.545146+0000","last_became_active":"2026-02-25T08:41:31.548939+0000","last_became_peered":"2026-02-25T08:41:31.548939+0000","last_unstale":"2026-02-25T08:41:41.545146+0000","last_undegraded":"2026-02-25T08:41:41.545146+0000","last_fullsized":"2026-02-25T08:41:41.545146+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":13092282,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":5144,"num_write":33,"num_write_kb":9391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.540502+0000","last_change":"2026-02-25T08:41:26.377253+0000","last_active":"2026-02-25T08:41:41.540502+0000","last_peered":"2026-02-25T08:41:41.540502+0000","last_clean":"2026-02-25T08:41:41.540502+0000","last_became_active":"2026-02-25T08:41:24.409565+0000","last_became_peered":"2026-02-25T08:41:24.409565+0000","last_unstale":"2026-02-25T08:41:41.540502+0000","last_undegraded":"2026-02-25T08:41:41.540502+0000","last_fullsized":"2026-02-25T08:41:41.540502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"22'22","reported_seq":73,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.546289+0000","last_change":"2026-02-25T08:41:41.549975+0000","last_active":"2026-02-25T08:41:41.546289+0000","last_peered":"2026-02-25T08:41:41.546289+0000","last_clean":"2026-02-25T08:41:41.546289+0000","last_became_active":"2026-02-25T08:41:31.586165+0000","last_became_peered":"2026-02-25T08:41:31.586165+0000","last_unstale":"2026-02-25T08:41:41.546289+0000","last_undegraded":"2026-02-25T08:41:41.546289+0000","last_fullsized":"2026-02-25T08:41:41.546289+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":11913304,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":11138,"num_write":25,"num_write_kb":6537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.540962+0000","last_change":"2026-02-25T08:41:26.377529+0000","last_active":"2026-02-25T08:41:41.540962+0000","last_peered":"2026-02-25T08:41:41.540962+0000","last_clean":"2026-02-25T08:41:41.540962+0000","last_became_active":"2026-02-25T08:41:24.426559+0000","last_became_peered":"2026-02-25T08:41:24.426559+0000","last_unstale":"2026-02-25T08:41:41.540962+0000","last_undegraded":"2026-02-25T08:41:41.540962+0000","last_fullsized":"2026-02-25T08:41:41.540962+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":112047884,"num_objects":52,"num_object_clones":4,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":202,"num_read_kb":73680,"num_write":251,"num_write_kb":66983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214748299264,"available":128636166144,"internally_reserved":0,"allocated":86112133120,"data_stored":86112133120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32148271104,"internally_reserved":0,"allocated":21538803712,"data_stored":21538803712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96468242432,"internally_reserved":0,"allocated":64592982016,"data_stored":64592982016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21033980,"kb_used_data":21033980,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31394804,"statfs":{"total":53687074816,"available":32148279296,"internally_reserved":0,"allocated":21538795520,"data_stored":21538795520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21031340,"kb_used_data":21031340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31397444,"statfs":{"total":53687074816,"available":32150982656,"internally_reserved":0,"allocated":21536092160,"data_stored":21536092160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21013748,"kb_used_data":21013748,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31415036,"statfs":{"total":53687074816,"available":32168996864,"internally_reserved":0,"allocated":21518077952,"data_stored":21518077952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21014796,"kb_used_data":21014796,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31413988,"statfs":{"total":53687074816,"available":32167923712,"internally_reserved":0,"allocated":21519151104,"data_stored":21519151104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32168988672,"internally_reserved":0,"allocated":21518086144,"data_stored":21518086144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32150982656,"internally_reserved":0,"allocated":21536092160,"data_stored":21536092160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32148271104,"internally_reserved":0,"allocated":21538803712,"data_stored":21538803712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32148271104,"internally_reserved":0,"allocated":21538803712,"data_stored":21538803712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53687074816,"available":32167923712,"internally_reserved":0,"allocated":21519151104,"data_stored":21519151104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53687074816,"available":32168988672,"internally_reserved":0,"allocated":21518086144,"data_stored":21518086144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53687074816,"available":32150982656,"internally_reserved":0,"allocated":21536092160,"data_stored":21536092160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53687074816,"available":32148271104,"internally_reserved":0,"allocated":21538803712,"data_stored":21538803712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:46.802 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:46.878 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:46.874+0000 7f7fef0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:46.880 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:46.879+0000 7f7fef0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:46.991 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:46.992 INFO:teuthology.orchestra.run.trial169.stderr:dumped all 2026-02-25T08:41:46.999 INFO:teuthology.orchestra.run.trial169.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2026-02-25T08:41:46.542621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112638271,"num_objects":56,"num_object_clones":4,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":255,"num_read_kb":73726,"num_write":332,"num_write_kb":68091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":263,"ondisk_log_size":263,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":84093864,"kb_used_data":84093864,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125621272,"statfs":{"total":214748299264,"available":128636182528,"internally_reserved":0,"allocated":86112116736,"data_stored":86112116736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015916"},"pg_stats":[{"pgid":"3.f","version":"23'11","reported_seq":63,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.546810+0000","last_change":"2026-02-25T08:41:41.550669+0000","last_active":"2026-02-25T08:41:41.546810+0000","last_peered":"2026-02-25T08:41:41.546810+0000","last_clean":"2026-02-25T08:41:41.546810+0000","last_became_active":"2026-02-25T08:41:31.577339+0000","last_became_peered":"2026-02-25T08:41:31.577339+0000","last_unstale":"2026-02-25T08:41:41.546810+0000","last_undegraded":"2026-02-25T08:41:41.546810+0000","last_fullsized":"2026-02-25T08:41:41.546810+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4319333,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":18,"num_read_kb":7817,"num_write":13,"num_write_kb":3115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":42,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.544890+0000","last_change":"2026-02-25T08:41:41.552033+0000","last_active":"2026-02-25T08:41:41.544890+0000","last_peered":"2026-02-25T08:41:41.544890+0000","last_clean":"2026-02-25T08:41:41.544890+0000","last_became_active":"2026-02-25T08:41:31.567579+0000","last_became_peered":"2026-02-25T08:41:31.567579+0000","last_unstale":"2026-02-25T08:41:41.544890+0000","last_undegraded":"2026-02-25T08:41:41.544890+0000","last_fullsized":"2026-02-25T08:41:41.544890+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'3","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.538452+0000","last_change":"2026-02-25T08:41:41.545268+0000","last_active":"2026-02-25T08:41:41.538452+0000","last_peered":"2026-02-25T08:41:41.538452+0000","last_clean":"2026-02-25T08:41:41.538452+0000","last_became_active":"2026-02-25T08:41:31.550562+0000","last_became_peered":"2026-02-25T08:41:31.550562+0000","last_unstale":"2026-02-25T08:41:41.538452+0000","last_undegraded":"2026-02-25T08:41:41.538452+0000","last_fullsized":"2026-02-25T08:41:41.538452+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2789929,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":9709,"num_write":4,"num_write_kb":1343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"20'16","reported_seq":66,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.543412+0000","last_change":"2026-02-25T08:41:41.549102+0000","last_active":"2026-02-25T08:41:41.543412+0000","last_peered":"2026-02-25T08:41:41.543412+0000","last_clean":"2026-02-25T08:41:41.543412+0000","last_became_active":"2026-02-25T08:41:31.571602+0000","last_became_peered":"2026-02-25T08:41:31.571602+0000","last_unstale":"2026-02-25T08:41:41.543412+0000","last_undegraded":"2026-02-25T08:41:41.543412+0000","last_fullsized":"2026-02-25T08:41:41.543412+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9966419,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":3790,"num_write":21,"num_write_kb":5481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"22'13","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.551408+0000","last_change":"2026-02-25T08:41:44.554216+0000","last_active":"2026-02-25T08:41:45.551408+0000","last_peered":"2026-02-25T08:41:45.551408+0000","last_clean":"2026-02-25T08:41:45.551408+0000","last_became_active":"2026-02-25T08:41:31.546602+0000","last_became_peered":"2026-02-25T08:41:31.546602+0000","last_unstale":"2026-02-25T08:41:45.551408+0000","last_undegraded":"2026-02-25T08:41:45.551408+0000","last_fullsized":"2026-02-25T08:41:45.551408+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8100875,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":5693,"num_write":17,"num_write_kb":4612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"22'22","reported_seq":71,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.541238+0000","last_change":"2026-02-25T08:41:41.546892+0000","last_active":"2026-02-25T08:41:41.541238+0000","last_peered":"2026-02-25T08:41:41.541238+0000","last_clean":"2026-02-25T08:41:41.541238+0000","last_became_active":"2026-02-25T08:41:31.576273+0000","last_became_peered":"2026-02-25T08:41:31.576273+0000","last_unstale":"2026-02-25T08:41:41.541238+0000","last_undegraded":"2026-02-25T08:41:41.541238+0000","last_fullsized":"2026-02-25T08:41:41.541238+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":9714314,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":7567,"num_write":25,"num_write_kb":6863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"22'11","reported_seq":56,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.542656+0000","last_change":"2026-02-25T08:41:41.548787+0000","last_active":"2026-02-25T08:41:41.542656+0000","last_peered":"2026-02-25T08:41:41.542656+0000","last_clean":"2026-02-25T08:41:41.542656+0000","last_became_active":"2026-02-25T08:41:31.569266+0000","last_became_peered":"2026-02-25T08:41:31.569266+0000","last_unstale":"2026-02-25T08:41:41.542656+0000","last_undegraded":"2026-02-25T08:41:41.542656+0000","last_fullsized":"2026-02-25T08:41:41.542656+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2873964,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":14,"num_write_kb":3511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"20'3","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.551054+0000","last_change":"2026-02-25T08:41:44.552340+0000","last_active":"2026-02-25T08:41:45.551054+0000","last_peered":"2026-02-25T08:41:45.551054+0000","last_clean":"2026-02-25T08:41:45.551054+0000","last_became_active":"2026-02-25T08:41:31.559832+0000","last_became_peered":"2026-02-25T08:41:31.559832+0000","last_unstale":"2026-02-25T08:41:45.551054+0000","last_undegraded":"2026-02-25T08:41:45.551054+0000","last_fullsized":"2026-02-25T08:41:45.551054+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151263,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3649,"num_write":4,"num_write_kb":1027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.541022+0000","last_change":"2026-02-25T08:41:26.373872+0000","last_active":"2026-02-25T08:41:41.541022+0000","last_peered":"2026-02-25T08:41:41.541022+0000","last_clean":"2026-02-25T08:41:41.541022+0000","last_became_active":"2026-02-25T08:41:24.399215+0000","last_became_peered":"2026-02-25T08:41:24.399215+0000","last_unstale":"2026-02-25T08:41:41.541022+0000","last_undegraded":"2026-02-25T08:41:41.541022+0000","last_fullsized":"2026-02-25T08:41:41.541022+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.551093+0000","last_change":"2026-02-25T08:41:44.553752+0000","last_active":"2026-02-25T08:41:45.551093+0000","last_peered":"2026-02-25T08:41:45.551093+0000","last_clean":"2026-02-25T08:41:45.551093+0000","last_became_active":"2026-02-25T08:41:31.602407+0000","last_became_peered":"2026-02-25T08:41:31.602407+0000","last_unstale":"2026-02-25T08:41:45.551093+0000","last_undegraded":"2026-02-25T08:41:45.551093+0000","last_fullsized":"2026-02-25T08:41:45.551093+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.542652+0000","last_change":"2026-02-25T08:41:26.374903+0000","last_active":"2026-02-25T08:41:41.542652+0000","last_peered":"2026-02-25T08:41:41.542652+0000","last_clean":"2026-02-25T08:41:41.542652+0000","last_became_active":"2026-02-25T08:41:24.402449+0000","last_became_peered":"2026-02-25T08:41:24.402449+0000","last_unstale":"2026-02-25T08:41:41.542652+0000","last_undegraded":"2026-02-25T08:41:41.542652+0000","last_fullsized":"2026-02-25T08:41:41.542652+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"20'12","reported_seq":63,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.546271+0000","last_change":"2026-02-25T08:41:41.548708+0000","last_active":"2026-02-25T08:41:41.546271+0000","last_peered":"2026-02-25T08:41:41.546271+0000","last_clean":"2026-02-25T08:41:41.546271+0000","last_became_active":"2026-02-25T08:41:31.591938+0000","last_became_peered":"2026-02-25T08:41:31.591938+0000","last_unstale":"2026-02-25T08:41:41.546271+0000","last_undegraded":"2026-02-25T08:41:41.546271+0000","last_fullsized":"2026-02-25T08:41:41.546271+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":4734,"num_write":16,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":54,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.544514+0000","last_change":"2026-02-25T08:41:26.374022+0000","last_active":"2026-02-25T08:41:41.544514+0000","last_peered":"2026-02-25T08:41:41.544514+0000","last_clean":"2026-02-25T08:41:41.544514+0000","last_became_active":"2026-02-25T08:41:24.420798+0000","last_became_peered":"2026-02-25T08:41:24.420798+0000","last_unstale":"2026-02-25T08:41:41.544514+0000","last_undegraded":"2026-02-25T08:41:41.544514+0000","last_fullsized":"2026-02-25T08:41:41.544514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"20'9","reported_seq":57,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.542344+0000","last_change":"2026-02-25T08:41:41.546520+0000","last_active":"2026-02-25T08:41:41.542344+0000","last_peered":"2026-02-25T08:41:41.542344+0000","last_clean":"2026-02-25T08:41:41.542344+0000","last_became_active":"2026-02-25T08:41:31.623233+0000","last_became_peered":"2026-02-25T08:41:31.623233+0000","last_unstale":"2026-02-25T08:41:41.542344+0000","last_undegraded":"2026-02-25T08:41:41.542344+0000","last_fullsized":"2026-02-25T08:41:41.542344+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840350,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":3971,"num_write":12,"num_write_kb":2773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.550121+0000","last_change":"2026-02-25T08:41:26.371067+0000","last_active":"2026-02-25T08:41:45.550121+0000","last_peered":"2026-02-25T08:41:45.550121+0000","last_clean":"2026-02-25T08:41:45.550121+0000","last_became_active":"2026-02-25T08:41:24.403313+0000","last_became_peered":"2026-02-25T08:41:24.403313+0000","last_unstale":"2026-02-25T08:41:45.550121+0000","last_undegraded":"2026-02-25T08:41:45.550121+0000","last_fullsized":"2026-02-25T08:41:45.550121+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":118,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.541622+0000","last_change":"2026-02-25T08:41:24.427616+0000","last_active":"2026-02-25T08:41:41.541622+0000","last_peered":"2026-02-25T08:41:41.541622+0000","last_clean":"2026-02-25T08:41:41.541622+0000","last_became_active":"2026-02-25T08:41:24.406166+0000","last_became_peered":"2026-02-25T08:41:24.406166+0000","last_unstale":"2026-02-25T08:41:41.541622+0000","last_undegraded":"2026-02-25T08:41:41.541622+0000","last_fullsized":"2026-02-25T08:41:41.541622+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"25'19","reported_seq":67,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.562383+0000","last_change":"2026-02-25T08:41:41.564290+0000","last_active":"2026-02-25T08:41:41.562383+0000","last_peered":"2026-02-25T08:41:41.562383+0000","last_clean":"2026-02-25T08:41:41.562383+0000","last_became_active":"2026-02-25T08:41:31.558932+0000","last_became_peered":"2026-02-25T08:41:31.558932+0000","last_unstale":"2026-02-25T08:41:41.562383+0000","last_undegraded":"2026-02-25T08:41:41.562383+0000","last_fullsized":"2026-02-25T08:41:41.562383+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7899191,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":6448,"num_write":21,"num_write_kb":5371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"25'28","reported_seq":76,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.559572+0000","last_change":"2026-02-25T08:41:41.561348+0000","last_active":"2026-02-25T08:41:41.559572+0000","last_peered":"2026-02-25T08:41:41.559572+0000","last_clean":"2026-02-25T08:41:41.559572+0000","last_became_active":"2026-02-25T08:41:31.602591+0000","last_became_peered":"2026-02-25T08:41:31.602591+0000","last_unstale":"2026-02-25T08:41:41.559572+0000","last_undegraded":"2026-02-25T08:41:41.559572+0000","last_fullsized":"2026-02-25T08:41:41.559572+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17445435,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":4014,"num_write":33,"num_write_kb":9199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.550932+0000","last_change":"2026-02-25T08:41:26.371743+0000","last_active":"2026-02-25T08:41:45.550932+0000","last_peered":"2026-02-25T08:41:45.550932+0000","last_clean":"2026-02-25T08:41:45.550932+0000","last_became_active":"2026-02-25T08:41:24.408398+0000","last_became_peered":"2026-02-25T08:41:24.408398+0000","last_unstale":"2026-02-25T08:41:45.550932+0000","last_undegraded":"2026-02-25T08:41:45.550932+0000","last_fullsized":"2026-02-25T08:41:45.550932+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'10","reported_seq":55,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.541375+0000","last_change":"2026-02-25T08:41:41.548902+0000","last_active":"2026-02-25T08:41:41.541375+0000","last_peered":"2026-02-25T08:41:41.541375+0000","last_clean":"2026-02-25T08:41:41.541375+0000","last_became_active":"2026-02-25T08:41:31.559738+0000","last_became_peered":"2026-02-25T08:41:31.559738+0000","last_unstale":"2026-02-25T08:41:41.541375+0000","last_undegraded":"2026-02-25T08:41:41.541375+0000","last_fullsized":"2026-02-25T08:41:41.541375+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5126069,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":3214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.548298+0000","last_change":"2026-02-25T08:41:26.375854+0000","last_active":"2026-02-25T08:41:41.548298+0000","last_peered":"2026-02-25T08:41:41.548298+0000","last_clean":"2026-02-25T08:41:41.548298+0000","last_became_active":"2026-02-25T08:41:24.420122+0000","last_became_peered":"2026-02-25T08:41:24.420122+0000","last_unstale":"2026-02-25T08:41:41.548298+0000","last_undegraded":"2026-02-25T08:41:41.548298+0000","last_fullsized":"2026-02-25T08:41:41.548298+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"26'29","reported_seq":78,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.545146+0000","last_change":"2026-02-25T08:41:41.552766+0000","last_active":"2026-02-25T08:41:41.545146+0000","last_peered":"2026-02-25T08:41:41.545146+0000","last_clean":"2026-02-25T08:41:41.545146+0000","last_became_active":"2026-02-25T08:41:31.548939+0000","last_became_peered":"2026-02-25T08:41:31.548939+0000","last_unstale":"2026-02-25T08:41:41.545146+0000","last_undegraded":"2026-02-25T08:41:41.545146+0000","last_fullsized":"2026-02-25T08:41:41.545146+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":13092282,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":5144,"num_write":33,"num_write_kb":9391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.540502+0000","last_change":"2026-02-25T08:41:26.377253+0000","last_active":"2026-02-25T08:41:41.540502+0000","last_peered":"2026-02-25T08:41:41.540502+0000","last_clean":"2026-02-25T08:41:41.540502+0000","last_became_active":"2026-02-25T08:41:24.409565+0000","last_became_peered":"2026-02-25T08:41:24.409565+0000","last_unstale":"2026-02-25T08:41:41.540502+0000","last_undegraded":"2026-02-25T08:41:41.540502+0000","last_fullsized":"2026-02-25T08:41:41.540502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"22'22","reported_seq":73,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.546289+0000","last_change":"2026-02-25T08:41:41.549975+0000","last_active":"2026-02-25T08:41:41.546289+0000","last_peered":"2026-02-25T08:41:41.546289+0000","last_clean":"2026-02-25T08:41:41.546289+0000","last_became_active":"2026-02-25T08:41:31.586165+0000","last_became_peered":"2026-02-25T08:41:31.586165+0000","last_unstale":"2026-02-25T08:41:41.546289+0000","last_undegraded":"2026-02-25T08:41:41.546289+0000","last_fullsized":"2026-02-25T08:41:41.546289+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":11913304,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":11138,"num_write":25,"num_write_kb":6537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.540962+0000","last_change":"2026-02-25T08:41:26.377529+0000","last_active":"2026-02-25T08:41:41.540962+0000","last_peered":"2026-02-25T08:41:41.540962+0000","last_clean":"2026-02-25T08:41:41.540962+0000","last_became_active":"2026-02-25T08:41:24.426559+0000","last_became_peered":"2026-02-25T08:41:24.426559+0000","last_unstale":"2026-02-25T08:41:41.540962+0000","last_undegraded":"2026-02-25T08:41:41.540962+0000","last_fullsized":"2026-02-25T08:41:41.540962+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":112047884,"num_objects":52,"num_object_clones":4,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":202,"num_read_kb":73680,"num_write":251,"num_write_kb":66983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214748299264,"available":128636166144,"internally_reserved":0,"allocated":86112133120,"data_stored":86112133120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32148271104,"internally_reserved":0,"allocated":21538803712,"data_stored":21538803712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96468242432,"internally_reserved":0,"allocated":64592982016,"data_stored":64592982016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21033980,"kb_used_data":21033980,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31394804,"statfs":{"total":53687074816,"available":32148279296,"internally_reserved":0,"allocated":21538795520,"data_stored":21538795520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21031340,"kb_used_data":21031340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31397444,"statfs":{"total":53687074816,"available":32150982656,"internally_reserved":0,"allocated":21536092160,"data_stored":21536092160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21013748,"kb_used_data":21013748,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31415036,"statfs":{"total":53687074816,"available":32168996864,"internally_reserved":0,"allocated":21518077952,"data_stored":21518077952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21014796,"kb_used_data":21014796,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31413988,"statfs":{"total":53687074816,"available":32167923712,"internally_reserved":0,"allocated":21519151104,"data_stored":21519151104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32168988672,"internally_reserved":0,"allocated":21518086144,"data_stored":21518086144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32150982656,"internally_reserved":0,"allocated":21536092160,"data_stored":21536092160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32148271104,"internally_reserved":0,"allocated":21538803712,"data_stored":21538803712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32148271104,"internally_reserved":0,"allocated":21538803712,"data_stored":21538803712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53687074816,"available":32167923712,"internally_reserved":0,"allocated":21519151104,"data_stored":21519151104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53687074816,"available":32168988672,"internally_reserved":0,"allocated":21518086144,"data_stored":21518086144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53687074816,"available":32150982656,"internally_reserved":0,"allocated":21536092160,"data_stored":21536092160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53687074816,"available":32148271104,"internally_reserved":0,"allocated":21538803712,"data_stored":21538803712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:47.002 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2026-02-25T08:41:47.002 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-25T08:41:47.002 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:41:47.078 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:47.074+0000 7f9ba0ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:47.080 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:47.078+0000 7f9ba0ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:47.192 INFO:teuthology.orchestra.run.trial169.stdout: 2026-02-25T08:41:47.192 INFO:teuthology.orchestra.run.trial169.stderr:dumped all 2026-02-25T08:41:47.201 INFO:teuthology.orchestra.run.trial169.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2026-02-25T08:41:46.542621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":112638271,"num_objects":56,"num_object_clones":4,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":255,"num_read_kb":73726,"num_write":332,"num_write_kb":68091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":263,"ondisk_log_size":263,"up":75,"acting":75,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":0,"num_osds":4,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":209715136,"kb_used":84093864,"kb_used_data":84093864,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":125621272,"statfs":{"total":214748299264,"available":128636182528,"internally_reserved":0,"allocated":86112116736,"data_stored":86112116736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015916"},"pg_stats":[{"pgid":"3.f","version":"23'11","reported_seq":63,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.546810+0000","last_change":"2026-02-25T08:41:41.550669+0000","last_active":"2026-02-25T08:41:41.546810+0000","last_peered":"2026-02-25T08:41:41.546810+0000","last_clean":"2026-02-25T08:41:41.546810+0000","last_became_active":"2026-02-25T08:41:31.577339+0000","last_became_peered":"2026-02-25T08:41:31.577339+0000","last_unstale":"2026-02-25T08:41:41.546810+0000","last_undegraded":"2026-02-25T08:41:41.546810+0000","last_fullsized":"2026-02-25T08:41:41.546810+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4319333,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":18,"num_read_kb":7817,"num_write":13,"num_write_kb":3115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":42,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.544890+0000","last_change":"2026-02-25T08:41:41.552033+0000","last_active":"2026-02-25T08:41:41.544890+0000","last_peered":"2026-02-25T08:41:41.544890+0000","last_clean":"2026-02-25T08:41:41.544890+0000","last_became_active":"2026-02-25T08:41:31.567579+0000","last_became_peered":"2026-02-25T08:41:31.567579+0000","last_unstale":"2026-02-25T08:41:41.544890+0000","last_undegraded":"2026-02-25T08:41:41.544890+0000","last_fullsized":"2026-02-25T08:41:41.544890+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"20'3","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.538452+0000","last_change":"2026-02-25T08:41:41.545268+0000","last_active":"2026-02-25T08:41:41.538452+0000","last_peered":"2026-02-25T08:41:41.538452+0000","last_clean":"2026-02-25T08:41:41.538452+0000","last_became_active":"2026-02-25T08:41:31.550562+0000","last_became_peered":"2026-02-25T08:41:31.550562+0000","last_unstale":"2026-02-25T08:41:41.538452+0000","last_undegraded":"2026-02-25T08:41:41.538452+0000","last_fullsized":"2026-02-25T08:41:41.538452+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2789929,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":9709,"num_write":4,"num_write_kb":1343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"20'16","reported_seq":66,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.543412+0000","last_change":"2026-02-25T08:41:41.549102+0000","last_active":"2026-02-25T08:41:41.543412+0000","last_peered":"2026-02-25T08:41:41.543412+0000","last_clean":"2026-02-25T08:41:41.543412+0000","last_became_active":"2026-02-25T08:41:31.571602+0000","last_became_peered":"2026-02-25T08:41:31.571602+0000","last_unstale":"2026-02-25T08:41:41.543412+0000","last_undegraded":"2026-02-25T08:41:41.543412+0000","last_fullsized":"2026-02-25T08:41:41.543412+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9966419,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":3790,"num_write":21,"num_write_kb":5481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"22'13","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.551408+0000","last_change":"2026-02-25T08:41:44.554216+0000","last_active":"2026-02-25T08:41:45.551408+0000","last_peered":"2026-02-25T08:41:45.551408+0000","last_clean":"2026-02-25T08:41:45.551408+0000","last_became_active":"2026-02-25T08:41:31.546602+0000","last_became_peered":"2026-02-25T08:41:31.546602+0000","last_unstale":"2026-02-25T08:41:45.551408+0000","last_undegraded":"2026-02-25T08:41:45.551408+0000","last_fullsized":"2026-02-25T08:41:45.551408+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8100875,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":5693,"num_write":17,"num_write_kb":4612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"22'22","reported_seq":71,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.541238+0000","last_change":"2026-02-25T08:41:41.546892+0000","last_active":"2026-02-25T08:41:41.541238+0000","last_peered":"2026-02-25T08:41:41.541238+0000","last_clean":"2026-02-25T08:41:41.541238+0000","last_became_active":"2026-02-25T08:41:31.576273+0000","last_became_peered":"2026-02-25T08:41:31.576273+0000","last_unstale":"2026-02-25T08:41:41.541238+0000","last_undegraded":"2026-02-25T08:41:41.541238+0000","last_fullsized":"2026-02-25T08:41:41.541238+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0,"stat_sum":{"num_bytes":9714314,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":7567,"num_write":25,"num_write_kb":6863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"22'11","reported_seq":56,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.542656+0000","last_change":"2026-02-25T08:41:41.548787+0000","last_active":"2026-02-25T08:41:41.542656+0000","last_peered":"2026-02-25T08:41:41.542656+0000","last_clean":"2026-02-25T08:41:41.542656+0000","last_became_active":"2026-02-25T08:41:31.569266+0000","last_became_peered":"2026-02-25T08:41:31.569266+0000","last_unstale":"2026-02-25T08:41:41.542656+0000","last_undegraded":"2026-02-25T08:41:41.542656+0000","last_fullsized":"2026-02-25T08:41:41.542656+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2873964,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":14,"num_write_kb":3511,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"20'3","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.551054+0000","last_change":"2026-02-25T08:41:44.552340+0000","last_active":"2026-02-25T08:41:45.551054+0000","last_peered":"2026-02-25T08:41:45.551054+0000","last_clean":"2026-02-25T08:41:45.551054+0000","last_became_active":"2026-02-25T08:41:31.559832+0000","last_became_peered":"2026-02-25T08:41:31.559832+0000","last_unstale":"2026-02-25T08:41:45.551054+0000","last_undegraded":"2026-02-25T08:41:45.551054+0000","last_fullsized":"2026-02-25T08:41:45.551054+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2151263,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":3649,"num_write":4,"num_write_kb":1027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.541022+0000","last_change":"2026-02-25T08:41:26.373872+0000","last_active":"2026-02-25T08:41:41.541022+0000","last_peered":"2026-02-25T08:41:41.541022+0000","last_clean":"2026-02-25T08:41:41.541022+0000","last_became_active":"2026-02-25T08:41:24.399215+0000","last_became_peered":"2026-02-25T08:41:24.399215+0000","last_unstale":"2026-02-25T08:41:41.541022+0000","last_undegraded":"2026-02-25T08:41:41.541022+0000","last_fullsized":"2026-02-25T08:41:41.541022+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.551093+0000","last_change":"2026-02-25T08:41:44.553752+0000","last_active":"2026-02-25T08:41:45.551093+0000","last_peered":"2026-02-25T08:41:45.551093+0000","last_clean":"2026-02-25T08:41:45.551093+0000","last_became_active":"2026-02-25T08:41:31.602407+0000","last_became_peered":"2026-02-25T08:41:31.602407+0000","last_unstale":"2026-02-25T08:41:45.551093+0000","last_undegraded":"2026-02-25T08:41:45.551093+0000","last_fullsized":"2026-02-25T08:41:45.551093+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.542652+0000","last_change":"2026-02-25T08:41:26.374903+0000","last_active":"2026-02-25T08:41:41.542652+0000","last_peered":"2026-02-25T08:41:41.542652+0000","last_clean":"2026-02-25T08:41:41.542652+0000","last_became_active":"2026-02-25T08:41:24.402449+0000","last_became_peered":"2026-02-25T08:41:24.402449+0000","last_unstale":"2026-02-25T08:41:41.542652+0000","last_undegraded":"2026-02-25T08:41:41.542652+0000","last_fullsized":"2026-02-25T08:41:41.542652+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"20'12","reported_seq":63,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.546271+0000","last_change":"2026-02-25T08:41:41.548708+0000","last_active":"2026-02-25T08:41:41.546271+0000","last_peered":"2026-02-25T08:41:41.546271+0000","last_clean":"2026-02-25T08:41:41.546271+0000","last_became_active":"2026-02-25T08:41:31.591938+0000","last_became_peered":"2026-02-25T08:41:31.591938+0000","last_unstale":"2026-02-25T08:41:41.546271+0000","last_undegraded":"2026-02-25T08:41:41.546271+0000","last_fullsized":"2026-02-25T08:41:41.546271+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9815156,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":4734,"num_write":16,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"15'2","reported_seq":54,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.544514+0000","last_change":"2026-02-25T08:41:26.374022+0000","last_active":"2026-02-25T08:41:41.544514+0000","last_peered":"2026-02-25T08:41:41.544514+0000","last_clean":"2026-02-25T08:41:41.544514+0000","last_became_active":"2026-02-25T08:41:24.420798+0000","last_became_peered":"2026-02-25T08:41:24.420798+0000","last_unstale":"2026-02-25T08:41:41.544514+0000","last_undegraded":"2026-02-25T08:41:41.544514+0000","last_fullsized":"2026-02-25T08:41:41.544514+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"20'9","reported_seq":57,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.542344+0000","last_change":"2026-02-25T08:41:41.546520+0000","last_active":"2026-02-25T08:41:41.542344+0000","last_peered":"2026-02-25T08:41:41.542344+0000","last_clean":"2026-02-25T08:41:41.542344+0000","last_became_active":"2026-02-25T08:41:31.623233+0000","last_became_peered":"2026-02-25T08:41:31.623233+0000","last_unstale":"2026-02-25T08:41:41.542344+0000","last_undegraded":"2026-02-25T08:41:41.542344+0000","last_fullsized":"2026-02-25T08:41:41.542344+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6840350,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":3971,"num_write":12,"num_write_kb":2773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"13'1","reported_seq":65,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.550121+0000","last_change":"2026-02-25T08:41:26.371067+0000","last_active":"2026-02-25T08:41:45.550121+0000","last_peered":"2026-02-25T08:41:45.550121+0000","last_clean":"2026-02-25T08:41:45.550121+0000","last_became_active":"2026-02-25T08:41:24.403313+0000","last_became_peered":"2026-02-25T08:41:24.403313+0000","last_unstale":"2026-02-25T08:41:45.550121+0000","last_undegraded":"2026-02-25T08:41:45.550121+0000","last_fullsized":"2026-02-25T08:41:45.550121+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"13'52","reported_seq":118,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.541622+0000","last_change":"2026-02-25T08:41:24.427616+0000","last_active":"2026-02-25T08:41:41.541622+0000","last_peered":"2026-02-25T08:41:41.541622+0000","last_clean":"2026-02-25T08:41:41.541622+0000","last_became_active":"2026-02-25T08:41:24.406166+0000","last_became_peered":"2026-02-25T08:41:24.406166+0000","last_unstale":"2026-02-25T08:41:41.541622+0000","last_undegraded":"2026-02-25T08:41:41.541622+0000","last_fullsized":"2026-02-25T08:41:41.541622+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"25'19","reported_seq":67,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.562383+0000","last_change":"2026-02-25T08:41:41.564290+0000","last_active":"2026-02-25T08:41:41.562383+0000","last_peered":"2026-02-25T08:41:41.562383+0000","last_clean":"2026-02-25T08:41:41.562383+0000","last_became_active":"2026-02-25T08:41:31.558932+0000","last_became_peered":"2026-02-25T08:41:31.558932+0000","last_unstale":"2026-02-25T08:41:41.562383+0000","last_undegraded":"2026-02-25T08:41:41.562383+0000","last_fullsized":"2026-02-25T08:41:41.562383+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":7899191,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":1,"num_read":17,"num_read_kb":6448,"num_write":21,"num_write_kb":5371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.6","version":"25'28","reported_seq":76,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.559572+0000","last_change":"2026-02-25T08:41:41.561348+0000","last_active":"2026-02-25T08:41:41.559572+0000","last_peered":"2026-02-25T08:41:41.559572+0000","last_clean":"2026-02-25T08:41:41.559572+0000","last_became_active":"2026-02-25T08:41:31.602591+0000","last_became_peered":"2026-02-25T08:41:31.602591+0000","last_unstale":"2026-02-25T08:41:41.559572+0000","last_undegraded":"2026-02-25T08:41:41.559572+0000","last_fullsized":"2026-02-25T08:41:41.559572+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17445435,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":4014,"num_write":33,"num_write_kb":9199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2026-02-25T08:41:45.550932+0000","last_change":"2026-02-25T08:41:26.371743+0000","last_active":"2026-02-25T08:41:45.550932+0000","last_peered":"2026-02-25T08:41:45.550932+0000","last_clean":"2026-02-25T08:41:45.550932+0000","last_became_active":"2026-02-25T08:41:24.408398+0000","last_became_peered":"2026-02-25T08:41:24.408398+0000","last_unstale":"2026-02-25T08:41:45.550932+0000","last_undegraded":"2026-02-25T08:41:45.550932+0000","last_fullsized":"2026-02-25T08:41:45.550932+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"22'10","reported_seq":55,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.541375+0000","last_change":"2026-02-25T08:41:41.548902+0000","last_active":"2026-02-25T08:41:41.541375+0000","last_peered":"2026-02-25T08:41:41.541375+0000","last_clean":"2026-02-25T08:41:41.541375+0000","last_became_active":"2026-02-25T08:41:31.559738+0000","last_became_peered":"2026-02-25T08:41:31.559738+0000","last_unstale":"2026-02-25T08:41:41.541375+0000","last_undegraded":"2026-02-25T08:41:41.541375+0000","last_fullsized":"2026-02-25T08:41:41.541375+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5126069,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":13,"num_write_kb":3214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.548298+0000","last_change":"2026-02-25T08:41:26.375854+0000","last_active":"2026-02-25T08:41:41.548298+0000","last_peered":"2026-02-25T08:41:41.548298+0000","last_clean":"2026-02-25T08:41:41.548298+0000","last_became_active":"2026-02-25T08:41:24.420122+0000","last_became_peered":"2026-02-25T08:41:24.420122+0000","last_unstale":"2026-02-25T08:41:41.548298+0000","last_undegraded":"2026-02-25T08:41:41.548298+0000","last_fullsized":"2026-02-25T08:41:41.548298+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"26'29","reported_seq":78,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.545146+0000","last_change":"2026-02-25T08:41:41.552766+0000","last_active":"2026-02-25T08:41:41.545146+0000","last_peered":"2026-02-25T08:41:41.545146+0000","last_clean":"2026-02-25T08:41:41.545146+0000","last_became_active":"2026-02-25T08:41:31.548939+0000","last_became_peered":"2026-02-25T08:41:31.548939+0000","last_unstale":"2026-02-25T08:41:41.545146+0000","last_undegraded":"2026-02-25T08:41:41.545146+0000","last_fullsized":"2026-02-25T08:41:41.545146+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":13092282,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":5144,"num_write":33,"num_write_kb":9391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.540502+0000","last_change":"2026-02-25T08:41:26.377253+0000","last_active":"2026-02-25T08:41:41.540502+0000","last_peered":"2026-02-25T08:41:41.540502+0000","last_clean":"2026-02-25T08:41:41.540502+0000","last_became_active":"2026-02-25T08:41:24.409565+0000","last_became_peered":"2026-02-25T08:41:24.409565+0000","last_unstale":"2026-02-25T08:41:41.540502+0000","last_undegraded":"2026-02-25T08:41:41.540502+0000","last_fullsized":"2026-02-25T08:41:41.540502+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"22'22","reported_seq":73,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.546289+0000","last_change":"2026-02-25T08:41:41.549975+0000","last_active":"2026-02-25T08:41:41.546289+0000","last_peered":"2026-02-25T08:41:41.546289+0000","last_clean":"2026-02-25T08:41:41.546289+0000","last_became_active":"2026-02-25T08:41:31.586165+0000","last_became_peered":"2026-02-25T08:41:31.586165+0000","last_unstale":"2026-02-25T08:41:41.546289+0000","last_undegraded":"2026-02-25T08:41:41.546289+0000","last_fullsized":"2026-02-25T08:41:41.546289+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:30.484777+0000","last_clean_scrub_stamp":"2026-02-25T08:41:30.484777+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0,"stat_sum":{"num_bytes":11913304,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":11138,"num_write":25,"num_write_kb":6537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":52,"reported_epoch":27,"state":"active+clean","last_fresh":"2026-02-25T08:41:41.540962+0000","last_change":"2026-02-25T08:41:26.377529+0000","last_active":"2026-02-25T08:41:41.540962+0000","last_peered":"2026-02-25T08:41:41.540962+0000","last_clean":"2026-02-25T08:41:41.540962+0000","last_became_active":"2026-02-25T08:41:24.426559+0000","last_became_peered":"2026-02-25T08:41:24.426559+0000","last_unstale":"2026-02-25T08:41:41.540962+0000","last_undegraded":"2026-02-25T08:41:41.540962+0000","last_fullsized":"2026-02-25T08:41:41.540962+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-25T08:41:23.347770+0000","last_clean_scrub_stamp":"2026-02-25T08:41:23.347770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":112047884,"num_objects":52,"num_object_clones":4,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":2,"num_read":202,"num_read_kb":73680,"num_write":251,"num_write_kb":66983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":214748299264,"available":128636166144,"internally_reserved":0,"allocated":86112133120,"data_stored":86112133120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":48,"acting":48,"num_store_stats":4},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":46,"num_write":79,"num_write_kb":1106,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":53687074816,"available":32148271104,"internally_reserved":0,"allocated":21538803712,"data_stored":21538803712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52,"ondisk_log_size":52,"up":3,"acting":3,"num_store_stats":1},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":161061224448,"available":96468242432,"internally_reserved":0,"allocated":64592982016,"data_stored":64592982016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":24,"acting":24,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":11,"seq":47244640275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21033980,"kb_used_data":21033980,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31394804,"statfs":{"total":53687074816,"available":32148279296,"internally_reserved":0,"allocated":21538795520,"data_stored":21538795520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21031340,"kb_used_data":21031340,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31397444,"statfs":{"total":53687074816,"available":32150982656,"internally_reserved":0,"allocated":21536092160,"data_stored":21536092160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":10,"seq":42949672984,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21013748,"kb_used_data":21013748,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31415036,"statfs":{"total":53687074816,"available":32168996864,"internally_reserved":0,"allocated":21518077952,"data_stored":21518077952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":52428784,"kb_used":21014796,"kb_used_data":21014796,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":31413988,"statfs":{"total":53687074816,"available":32167923712,"internally_reserved":0,"allocated":21519151104,"data_stored":21519151104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":53687074816,"available":32168988672,"internally_reserved":0,"allocated":21518086144,"data_stored":21518086144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":53687074816,"available":32150982656,"internally_reserved":0,"allocated":21536092160,"data_stored":21536092160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":53687074816,"available":32148271104,"internally_reserved":0,"allocated":21538803712,"data_stored":21538803712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":53687074816,"available":32148271104,"internally_reserved":0,"allocated":21538803712,"data_stored":21538803712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":53687074816,"available":32167923712,"internally_reserved":0,"allocated":21519151104,"data_stored":21519151104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":53687074816,"available":32168988672,"internally_reserved":0,"allocated":21518086144,"data_stored":21518086144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":53687074816,"available":32150982656,"internally_reserved":0,"allocated":21536092160,"data_stored":21536092160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":53687074816,"available":32148271104,"internally_reserved":0,"allocated":21538803712,"data_stored":21538803712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-25T08:41:47.552 INFO:tasks.rados.rados.0.trial169.stdout:116: done (0 left) 2026-02-25T08:41:47.552 INFO:tasks.rados.rados.0.trial169.stdout:117: snap_remove snap 6 2026-02-25T08:41:48.556 INFO:tasks.rados.rados.0.trial169.stdout:117: done (0 left) 2026-02-25T08:41:48.557 INFO:tasks.rados.rados.0.trial169.stdout:118: write (excl) oid 26 current snap is 8 2026-02-25T08:41:48.557 INFO:tasks.rados.rados.0.trial169.stdout:118: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2026-02-25T08:41:48.561 INFO:tasks.rados.rados.0.trial169.stdout:118: writing trial16941936-26 from 460525 to 1151377 tid 1 2026-02-25T08:41:48.566 INFO:tasks.rados.rados.0.trial169.stdout:118: writing trial16941936-26 from 1622620 to 2352972 tid 2 2026-02-25T08:41:48.570 INFO:tasks.rados.rados.0.trial169.stdout:118: writing trial16941936-26 from 3060489 to 3260525 tid 3 2026-02-25T08:41:48.570 INFO:tasks.rados.rados.0.trial169.stdout:119: snap_remove snap 5 2026-02-25T08:41:48.737 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:41:48.813 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:48.811+0000 7f84ce4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:48.815 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:48.814+0000 7f84ce4cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:49.563 INFO:tasks.rados.rados.0.trial169.stdout:119: done (1 left) 2026-02-25T08:41:49.564 INFO:tasks.rados.rados.0.trial169.stdout:120: read oid 7 snap -1 2026-02-25T08:41:49.564 INFO:tasks.rados.rados.0.trial169.stdout:120: expect (ObjNum 6 snap 0 seq_num 6) 2026-02-25T08:41:49.564 INFO:tasks.rados.rados.0.trial169.stdout:121: delete oid 3 current snap is 8 2026-02-25T08:41:49.564 INFO:tasks.rados.rados.0.trial169.stdout:118: finishing write tid 1 to trial16941936-26 2026-02-25T08:41:49.564 INFO:tasks.rados.rados.0.trial169.stdout:118: finishing write tid 2 to trial16941936-26 2026-02-25T08:41:49.564 INFO:tasks.rados.rados.0.trial169.stdout:118: finishing write tid 3 to trial16941936-26 2026-02-25T08:41:49.564 INFO:tasks.rados.rados.0.trial169.stdout:118: finishing write tid 4 to trial16941936-26 2026-02-25T08:41:49.564 INFO:tasks.rados.rados.0.trial169.stdout:118: finishing write tid 5 to trial16941936-26 2026-02-25T08:41:49.564 INFO:tasks.rados.rados.0.trial169.stdout:118: oid 26 updating version 0 to 30 2026-02-25T08:41:49.564 INFO:tasks.rados.rados.0.trial169.stdout:118: oid 26 version 30 is already newer than 30 2026-02-25T08:41:49.565 INFO:tasks.rados.rados.0.trial169.stdout:118: oid 26 version 30 is already newer than 30 2026-02-25T08:41:49.565 INFO:tasks.rados.rados.0.trial169.stdout:118: oid 26 version 30 is already newer than 27 2026-02-25T08:41:49.565 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 26 v 30 (ObjNum 75 snap 8 seq_num 75) dirty exists 2026-02-25T08:41:49.565 INFO:tasks.rados.rados.0.trial169.stdout:118: left oid 26 (ObjNum 75 snap 8 seq_num 75) 2026-02-25T08:41:49.598 INFO:tasks.rados.rados.0.trial169.stdout:120: expect (ObjNum 6 snap 0 seq_num 6) 2026-02-25T08:41:49.599 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1631716 2026-02-25T08:41:49.607 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:49.644 INFO:tasks.rados.rados.0.trial169.stdout:118: done (2 left) 2026-02-25T08:41:49.644 INFO:tasks.rados.rados.0.trial169.stdout:120: done (1 left) 2026-02-25T08:41:49.644 INFO:tasks.rados.rados.0.trial169.stdout:121: done (0 left) 2026-02-25T08:41:49.644 INFO:tasks.rados.rados.0.trial169.stdout:122: snap_remove snap 7 2026-02-25T08:41:50.567 INFO:tasks.rados.rados.0.trial169.stdout:122: done (0 left) 2026-02-25T08:41:50.567 INFO:tasks.rados.rados.0.trial169.stdout:123: write (excl) oid 16 current snap is 8 2026-02-25T08:41:50.567 INFO:tasks.rados.rados.0.trial169.stdout:123: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2026-02-25T08:41:50.571 INFO:tasks.rados.rados.0.trial169.stdout:123: writing trial16941936-16 from 494139 to 1051842 tid 1 2026-02-25T08:41:50.576 INFO:tasks.rados.rados.0.trial169.stdout:123: writing trial16941936-16 from 1455937 to 2189958 tid 2 2026-02-25T08:41:50.580 INFO:tasks.rados.rados.0.trial169.stdout:123: writing trial16941936-16 from 2948041 to 3294139 tid 3 2026-02-25T08:41:50.581 INFO:tasks.rados.rados.0.trial169.stdout:124: write (excl) oid 25 current snap is 8 2026-02-25T08:41:50.581 INFO:tasks.rados.rados.0.trial169.stdout:124: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2026-02-25T08:41:50.585 INFO:tasks.rados.rados.0.trial169.stdout:124: writing trial16941936-25 from 510946 to 1143898 tid 1 2026-02-25T08:41:50.589 INFO:tasks.rados.rados.0.trial169.stdout:124: writing trial16941936-25 from 1798066 to 2275745 tid 2 2026-02-25T08:41:50.593 INFO:tasks.rados.rados.0.trial169.stdout:124: writing trial16941936-25 from 2742758 to 3199975 tid 3 2026-02-25T08:41:50.593 INFO:tasks.rados.rados.0.trial169.stdout:125: write (excl) oid 46 current snap is 8 2026-02-25T08:41:50.593 INFO:tasks.rados.rados.0.trial169.stdout:125: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2026-02-25T08:41:50.598 INFO:tasks.rados.rados.0.trial169.stdout:125: writing trial16941936-46 from 527753 to 1235954 tid 1 2026-02-25T08:41:50.603 INFO:tasks.rados.rados.0.trial169.stdout:125: writing trial16941936-46 from 1856548 to 2594238 tid 2 2026-02-25T08:41:50.606 INFO:tasks.rados.rados.0.trial169.stdout:125: writing trial16941936-46 from 3286534 to 3327753 tid 3 2026-02-25T08:41:50.606 INFO:tasks.rados.rados.0.trial169.stdout:126: read oid 43 snap -1 2026-02-25T08:41:50.607 INFO:tasks.rados.rados.0.trial169.stdout:126: expect (ObjNum 42 snap 0 seq_num 42) 2026-02-25T08:41:50.607 INFO:tasks.rados.rados.0.trial169.stdout:127: delete oid 20 current snap is 8 2026-02-25T08:41:50.607 INFO:tasks.rados.rados.0.trial169.stdout:123: finishing write tid 1 to trial16941936-16 2026-02-25T08:41:50.607 INFO:tasks.rados.rados.0.trial169.stdout:124: finishing write tid 1 to trial16941936-25 2026-02-25T08:41:50.619 INFO:tasks.rados.rados.0.trial169.stdout:125: finishing write tid 1 to trial16941936-46 2026-02-25T08:41:50.627 INFO:tasks.rados.rados.0.trial169.stdout:124: finishing write tid 2 to trial16941936-25 2026-02-25T08:41:50.632 INFO:tasks.rados.rados.0.trial169.stdout:126: expect (ObjNum 42 snap 0 seq_num 42) 2026-02-25T08:41:50.676 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:50.700 INFO:tasks.rados.rados.0.trial169.stdout:124: finishing write tid 3 to trial16941936-25 2026-02-25T08:41:50.700 INFO:tasks.rados.rados.0.trial169.stdout:123: finishing write tid 2 to trial16941936-16 2026-02-25T08:41:50.700 INFO:tasks.rados.rados.0.trial169.stdout:123: finishing write tid 3 to trial16941936-16 2026-02-25T08:41:50.700 INFO:tasks.rados.rados.0.trial169.stdout:124: finishing write tid 4 to trial16941936-25 2026-02-25T08:41:50.700 INFO:tasks.rados.rados.0.trial169.stdout:126: done (4 left) 2026-02-25T08:41:50.701 INFO:tasks.rados.rados.0.trial169.stdout:127: done (3 left) 2026-02-25T08:41:50.701 INFO:tasks.rados.rados.0.trial169.stdout:128: snap_create 2026-02-25T08:41:50.701 INFO:tasks.rados.rados.0.trial169.stdout:123: finishing write tid 4 to trial16941936-16 2026-02-25T08:41:50.701 INFO:tasks.rados.rados.0.trial169.stdout:124: finishing write tid 5 to trial16941936-25 2026-02-25T08:41:50.701 INFO:tasks.rados.rados.0.trial169.stdout:124: oid 25 updating version 0 to 17 2026-02-25T08:41:50.701 INFO:tasks.rados.rados.0.trial169.stdout:124: oid 25 version 17 is already newer than 17 2026-02-25T08:41:50.701 INFO:tasks.rados.rados.0.trial169.stdout:124: oid 25 version 17 is already newer than 17 2026-02-25T08:41:50.701 INFO:tasks.rados.rados.0.trial169.stdout:124: oid 25 version 17 is already newer than 14 2026-02-25T08:41:50.701 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 25 v 17 (ObjNum 78 snap 8 seq_num 78) dirty exists 2026-02-25T08:41:50.701 INFO:tasks.rados.rados.0.trial169.stdout:124: left oid 25 (ObjNum 78 snap 8 seq_num 78) 2026-02-25T08:41:50.701 INFO:tasks.rados.rados.0.trial169.stdout:123: finishing write tid 5 to trial16941936-16 2026-02-25T08:41:50.702 INFO:tasks.rados.rados.0.trial169.stdout:123: oid 16 updating version 0 to 10 2026-02-25T08:41:50.702 INFO:tasks.rados.rados.0.trial169.stdout:123: oid 16 updating version 10 to 13 2026-02-25T08:41:50.702 INFO:tasks.rados.rados.0.trial169.stdout:123: oid 16 version 13 is already newer than 13 2026-02-25T08:41:50.702 INFO:tasks.rados.rados.0.trial169.stdout:123: oid 16 version 13 is already newer than 13 2026-02-25T08:41:50.702 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 16 v 13 (ObjNum 77 snap 8 seq_num 77) dirty exists 2026-02-25T08:41:50.702 INFO:tasks.rados.rados.0.trial169.stdout:123: left oid 16 (ObjNum 77 snap 8 seq_num 77) 2026-02-25T08:41:50.702 INFO:tasks.rados.rados.0.trial169.stdout:125: finishing write tid 2 to trial16941936-46 2026-02-25T08:41:50.702 INFO:tasks.rados.rados.0.trial169.stdout:125: finishing write tid 3 to trial16941936-46 2026-02-25T08:41:50.702 INFO:tasks.rados.rados.0.trial169.stdout:125: finishing write tid 4 to trial16941936-46 2026-02-25T08:41:50.702 INFO:tasks.rados.rados.0.trial169.stdout:125: finishing write tid 5 to trial16941936-46 2026-02-25T08:41:50.702 INFO:tasks.rados.rados.0.trial169.stdout:125: oid 46 updating version 0 to 42 2026-02-25T08:41:50.702 INFO:tasks.rados.rados.0.trial169.stdout:125: oid 46 version 42 is already newer than 39 2026-02-25T08:41:50.702 INFO:tasks.rados.rados.0.trial169.stdout:125: oid 46 version 42 is already newer than 42 2026-02-25T08:41:50.703 INFO:tasks.rados.rados.0.trial169.stdout:125: oid 46 version 42 is already newer than 42 2026-02-25T08:41:50.703 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 46 v 42 (ObjNum 79 snap 8 seq_num 79) dirty exists 2026-02-25T08:41:50.703 INFO:tasks.rados.rados.0.trial169.stdout:125: left oid 46 (ObjNum 79 snap 8 seq_num 79) 2026-02-25T08:41:50.877 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:41:50.955 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:50.952+0000 7fe7beb64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:50.957 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:50.955+0000 7fe7beb64640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:51.572 INFO:tasks.rados.rados.0.trial169.stdout:123: done (3 left) 2026-02-25T08:41:51.572 INFO:tasks.rados.rados.0.trial169.stdout:124: done (2 left) 2026-02-25T08:41:51.572 INFO:tasks.rados.rados.0.trial169.stdout:125: done (1 left) 2026-02-25T08:41:51.572 INFO:tasks.rados.rados.0.trial169.stdout:128: done (0 left) 2026-02-25T08:41:51.572 INFO:tasks.rados.rados.0.trial169.stdout:129: read oid 1 snap -1 2026-02-25T08:41:51.572 INFO:tasks.rados.rados.0.trial169.stdout:129: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-25T08:41:51.572 INFO:tasks.rados.rados.0.trial169.stdout:130: write (excl) oid 11 current snap is 9 2026-02-25T08:41:51.573 INFO:tasks.rados.rados.0.trial169.stdout:130: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2026-02-25T08:41:51.575 INFO:tasks.rados.rados.0.trial169.stdout:130: writing trial16941936-11 from 561367 to 1020066 tid 1 2026-02-25T08:41:51.580 INFO:tasks.rados.rados.0.trial169.stdout:130: writing trial16941936-11 from 1573512 to 2314871 tid 2 2026-02-25T08:41:51.584 INFO:tasks.rados.rados.0.trial169.stdout:130: writing trial16941936-11 from 2941380 to 3361367 tid 3 2026-02-25T08:41:51.584 INFO:tasks.rados.rados.0.trial169.stdout:131: write oid 12 current snap is 9 2026-02-25T08:41:51.584 INFO:tasks.rados.rados.0.trial169.stdout:131: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2026-02-25T08:41:51.588 INFO:tasks.rados.rados.0.trial169.stdout:131: writing trial16941936-12 from 578174 to 1112122 tid 1 2026-02-25T08:41:51.591 INFO:tasks.rados.rados.0.trial169.stdout:131: writing trial16941936-12 from 1515641 to 2000658 tid 2 2026-02-25T08:41:51.596 INFO:tasks.rados.rados.0.trial169.stdout:131: writing trial16941936-12 from 2736097 to 3238402 tid 3 2026-02-25T08:41:51.597 INFO:tasks.rados.rados.0.trial169.stdout:132: write oid 50 current snap is 9 2026-02-25T08:41:51.597 INFO:tasks.rados.rados.0.trial169.stdout:132: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2026-02-25T08:41:51.601 INFO:tasks.rados.rados.0.trial169.stdout:132: writing trial16941936-50 from 594981 to 1204178 tid 1 2026-02-25T08:41:51.607 INFO:tasks.rados.rados.0.trial169.stdout:132: writing trial16941936-50 from 1974123 to 2719151 tid 2 2026-02-25T08:41:51.609 INFO:tasks.rados.rados.0.trial169.stdout:132: writing trial16941936-50 from 3279873 to 3394981 tid 3 2026-02-25T08:41:51.609 INFO:tasks.rados.rados.0.trial169.stdout:133: write oid 15 current snap is 9 2026-02-25T08:41:51.610 INFO:tasks.rados.rados.0.trial169.stdout:133: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2026-02-25T08:41:51.613 INFO:tasks.rados.rados.0.trial169.stdout:133: writing trial16941936-15 from 611788 to 1012587 tid 1 2026-02-25T08:41:51.618 INFO:tasks.rados.rados.0.trial169.stdout:133: writing trial16941936-15 from 1748958 to 2237644 tid 2 2026-02-25T08:41:51.622 INFO:tasks.rados.rados.0.trial169.stdout:133: writing trial16941936-15 from 2907296 to 3411788 tid 3 2026-02-25T08:41:51.622 INFO:tasks.rados.rados.0.trial169.stdout:134: snap_remove snap 8 2026-02-25T08:41:52.204 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3] 2026-02-25T08:41:52.204 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-25T08:41:52.205 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3] 2026-02-25T08:41:52.205 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-25T08:41:52.206 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:41:52.260 INFO:tasks.ceph.osd.0:Stopped 2026-02-25T08:41:52.573 INFO:tasks.rados.rados.0.trial169.stdout:134: done (5 left) 2026-02-25T08:41:52.573 INFO:tasks.rados.rados.0.trial169.stdout:135: delete oid 8 current snap is 9 2026-02-25T08:41:52.573 INFO:tasks.rados.rados.0.trial169.stdout:129: expect (ObjNum 0 snap 0 seq_num 0) 2026-02-25T08:41:52.573 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 328614 2026-02-25T08:41:52.590 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1189448 2026-02-25T08:41:52.599 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:41:52.618 INFO:tasks.rados.rados.0.trial169.stdout:130: finishing write tid 1 to trial16941936-11 2026-02-25T08:41:52.618 INFO:tasks.rados.rados.0.trial169.stdout:130: finishing write tid 2 to trial16941936-11 2026-02-25T08:41:52.618 INFO:tasks.rados.rados.0.trial169.stdout:130: finishing write tid 3 to trial16941936-11 2026-02-25T08:41:52.618 INFO:tasks.rados.rados.0.trial169.stdout:130: finishing write tid 4 to trial16941936-11 2026-02-25T08:41:52.618 INFO:tasks.rados.rados.0.trial169.stdout:130: finishing write tid 5 to trial16941936-11 2026-02-25T08:41:52.619 INFO:tasks.rados.rados.0.trial169.stdout:130: oid 11 updating version 0 to 18 2026-02-25T08:41:52.619 INFO:tasks.rados.rados.0.trial169.stdout:130: oid 11 version 18 is already newer than 15 2026-02-25T08:41:52.619 INFO:tasks.rados.rados.0.trial169.stdout:130: oid 11 version 18 is already newer than 18 2026-02-25T08:41:52.619 INFO:tasks.rados.rados.0.trial169.stdout:130: oid 11 version 18 is already newer than 18 2026-02-25T08:41:52.619 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 11 v 18 (ObjNum 81 snap 9 seq_num 81) dirty exists 2026-02-25T08:41:52.619 INFO:tasks.rados.rados.0.trial169.stdout:130: left oid 11 (ObjNum 81 snap 9 seq_num 81) 2026-02-25T08:41:52.619 INFO:tasks.rados.rados.0.trial169.stdout:131: finishing write tid 1 to trial16941936-12 2026-02-25T08:41:52.619 INFO:tasks.rados.rados.0.trial169.stdout:132: finishing write tid 1 to trial16941936-50 2026-02-25T08:41:52.619 INFO:tasks.rados.rados.0.trial169.stdout:131: finishing write tid 2 to trial16941936-12 2026-02-25T08:41:52.619 INFO:tasks.rados.rados.0.trial169.stdout:129: done (5 left) 2026-02-25T08:41:52.619 INFO:tasks.rados.rados.0.trial169.stdout:130: done (4 left) 2026-02-25T08:41:52.620 INFO:tasks.rados.rados.0.trial169.stdout:135: done (3 left) 2026-02-25T08:41:52.620 INFO:tasks.rados.rados.0.trial169.stdout:136: write (excl) oid 20 current snap is 9 2026-02-25T08:41:52.620 INFO:tasks.rados.rados.0.trial169.stdout:131: finishing write tid 3 to trial16941936-12 2026-02-25T08:41:52.620 INFO:tasks.rados.rados.0.trial169.stdout:132: finishing write tid 2 to trial16941936-50 2026-02-25T08:41:52.620 INFO:tasks.rados.rados.0.trial169.stdout:131: finishing write tid 4 to trial16941936-12 2026-02-25T08:41:52.620 INFO:tasks.rados.rados.0.trial169.stdout:131: finishing write tid 5 to trial16941936-12 2026-02-25T08:41:52.620 INFO:tasks.rados.rados.0.trial169.stdout:131: oid 12 updating version 0 to 15 2026-02-25T08:41:52.620 INFO:tasks.rados.rados.0.trial169.stdout:131: oid 12 version 15 is already newer than 15 2026-02-25T08:41:52.620 INFO:tasks.rados.rados.0.trial169.stdout:131: oid 12 version 15 is already newer than 15 2026-02-25T08:41:52.620 INFO:tasks.rados.rados.0.trial169.stdout:131: oid 12 version 15 is already newer than 15 2026-02-25T08:41:52.620 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 12 v 15 (ObjNum 82 snap 9 seq_num 82) dirty exists 2026-02-25T08:41:52.620 INFO:tasks.rados.rados.0.trial169.stdout:131: left oid 12 (ObjNum 82 snap 9 seq_num 82) 2026-02-25T08:41:52.620 INFO:tasks.rados.rados.0.trial169.stdout:132: finishing write tid 3 to trial16941936-50 2026-02-25T08:41:52.621 INFO:tasks.rados.rados.0.trial169.stdout:132: finishing write tid 4 to trial16941936-50 2026-02-25T08:41:52.621 INFO:tasks.rados.rados.0.trial169.stdout:132: finishing write tid 5 to trial16941936-50 2026-02-25T08:41:52.621 INFO:tasks.rados.rados.0.trial169.stdout:132: oid 50 updating version 0 to 43 2026-02-25T08:41:52.621 INFO:tasks.rados.rados.0.trial169.stdout:132: oid 50 version 43 is already newer than 43 2026-02-25T08:41:52.621 INFO:tasks.rados.rados.0.trial169.stdout:132: oid 50 version 43 is already newer than 43 2026-02-25T08:41:52.621 INFO:tasks.rados.rados.0.trial169.stdout:132: oid 50 version 43 is already newer than 43 2026-02-25T08:41:52.621 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 50 v 43 (ObjNum 83 snap 9 seq_num 83) dirty exists 2026-02-25T08:41:52.621 INFO:tasks.rados.rados.0.trial169.stdout:132: left oid 50 (ObjNum 83 snap 9 seq_num 83) 2026-02-25T08:41:52.621 INFO:tasks.rados.rados.0.trial169.stdout:133: finishing write tid 1 to trial16941936-15 2026-02-25T08:41:52.621 INFO:tasks.rados.rados.0.trial169.stdout:133: finishing write tid 2 to trial16941936-15 2026-02-25T08:41:52.621 INFO:tasks.rados.rados.0.trial169.stdout:133: finishing write tid 3 to trial16941936-15 2026-02-25T08:41:52.621 INFO:tasks.rados.rados.0.trial169.stdout:133: finishing write tid 4 to trial16941936-15 2026-02-25T08:41:52.622 INFO:tasks.rados.rados.0.trial169.stdout:133: finishing write tid 5 to trial16941936-15 2026-02-25T08:41:52.622 INFO:tasks.rados.rados.0.trial169.stdout:133: oid 15 updating version 0 to 18 2026-02-25T08:41:52.622 INFO:tasks.rados.rados.0.trial169.stdout:133: oid 15 version 18 is already newer than 18 2026-02-25T08:41:52.622 INFO:tasks.rados.rados.0.trial169.stdout:133: oid 15 version 18 is already newer than 18 2026-02-25T08:41:52.622 INFO:tasks.rados.rados.0.trial169.stdout:133: oid 15 version 18 is already newer than 18 2026-02-25T08:41:52.622 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 15 v 18 (ObjNum 84 snap 9 seq_num 84) dirty exists 2026-02-25T08:41:52.622 INFO:tasks.rados.rados.0.trial169.stdout:133: left oid 15 (ObjNum 84 snap 9 seq_num 84) 2026-02-25T08:41:52.622 INFO:tasks.rados.rados.0.trial169.stdout:136: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2026-02-25T08:41:52.623 INFO:tasks.rados.rados.0.trial169.stdout:136: writing trial16941936-20 from 645402 to 1196699 tid 1 2026-02-25T08:41:52.625 INFO:tasks.rados.rados.0.trial169.stdout:136: writing trial16941936-20 from 1749569 to 2241924 tid 2 2026-02-25T08:41:52.630 INFO:tasks.rados.rados.0.trial169.stdout:136: writing trial16941936-20 from 2845789 to 3445402 tid 3 2026-02-25T08:41:52.630 INFO:tasks.rados.rados.0.trial169.stdout:131: done (3 left) 2026-02-25T08:41:52.630 INFO:tasks.rados.rados.0.trial169.stdout:132: done (2 left) 2026-02-25T08:41:52.630 INFO:tasks.rados.rados.0.trial169.stdout:133: done (1 left) 2026-02-25T08:41:52.630 INFO:tasks.rados.rados.0.trial169.stdout:137: write (excl) oid 40 current snap is 9 2026-02-25T08:41:52.630 INFO:tasks.rados.rados.0.trial169.stdout:137: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2026-02-25T08:41:52.634 INFO:tasks.rados.rados.0.trial169.stdout:137: writing trial16941936-40 from 662209 to 1288755 tid 1 2026-02-25T08:41:52.639 INFO:tasks.rados.rados.0.trial169.stdout:137: writing trial16941936-40 from 1808051 to 2444064 tid 2 2026-02-25T08:41:52.643 INFO:tasks.rados.rados.0.trial169.stdout:137: writing trial16941936-40 from 2873212 to 3462209 tid 3 2026-02-25T08:41:52.643 INFO:tasks.rados.rados.0.trial169.stdout:138: delete oid 31 current snap is 9 2026-02-25T08:41:52.651 INFO:tasks.rados.rados.0.trial169.stdout:137: finishing write tid 1 to trial16941936-40 2026-02-25T08:41:52.654 INFO:tasks.rados.rados.0.trial169.stdout:138: done (2 left) 2026-02-25T08:41:52.654 INFO:tasks.rados.rados.0.trial169.stdout:139: write (excl) oid 30 current snap is 9 2026-02-25T08:41:52.654 INFO:tasks.rados.rados.0.trial169.stdout:139: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2026-02-25T08:41:52.659 INFO:tasks.rados.rados.0.trial169.stdout:139: writing trial16941936-30 from 695823 to 1472867 tid 1 2026-02-25T08:41:52.663 INFO:tasks.rados.rados.0.trial169.stdout:139: writing trial16941936-30 from 1925015 to 2564697 tid 2 2026-02-25T08:41:52.667 INFO:tasks.rados.rados.0.trial169.stdout:139: writing trial16941936-30 from 3328058 to 3495823 tid 3 2026-02-25T08:41:52.667 INFO:tasks.rados.rados.0.trial169.stdout:140: write oid 27 current snap is 9 2026-02-25T08:41:52.667 INFO:tasks.rados.rados.0.trial169.stdout:137: finishing write tid 2 to trial16941936-40 2026-02-25T08:41:52.667 INFO:tasks.rados.rados.0.trial169.stdout:140: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2026-02-25T08:41:52.671 INFO:tasks.rados.rados.0.trial169.stdout:140: writing trial16941936-27 from 712630 to 1164923 tid 1 2026-02-25T08:41:52.675 INFO:tasks.rados.rados.0.trial169.stdout:140: writing trial16941936-27 from 1583497 to 2083190 tid 2 2026-02-25T08:41:52.679 INFO:tasks.rados.rados.0.trial169.stdout:140: writing trial16941936-27 from 2555481 to 2980668 tid 3 2026-02-25T08:41:52.679 INFO:tasks.rados.rados.0.trial169.stdout:141: snap_create 2026-02-25T08:41:52.680 INFO:tasks.rados.rados.0.trial169.stdout:137: finishing write tid 3 to trial16941936-40 2026-02-25T08:41:52.680 INFO:tasks.rados.rados.0.trial169.stdout:137: finishing write tid 4 to trial16941936-40 2026-02-25T08:41:52.680 INFO:tasks.rados.rados.0.trial169.stdout:137: finishing write tid 5 to trial16941936-40 2026-02-25T08:41:52.680 INFO:tasks.rados.rados.0.trial169.stdout:137: oid 40 updating version 0 to 49 2026-02-25T08:41:52.680 INFO:tasks.rados.rados.0.trial169.stdout:137: oid 40 version 49 is already newer than 49 2026-02-25T08:41:52.680 INFO:tasks.rados.rados.0.trial169.stdout:137: oid 40 version 49 is already newer than 49 2026-02-25T08:41:52.680 INFO:tasks.rados.rados.0.trial169.stdout:137: oid 40 version 49 is already newer than 46 2026-02-25T08:41:52.680 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 40 v 49 (ObjNum 87 snap 9 seq_num 87) dirty exists 2026-02-25T08:41:52.680 INFO:tasks.rados.rados.0.trial169.stdout:137: left oid 40 (ObjNum 87 snap 9 seq_num 87) 2026-02-25T08:41:53.016 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:41:53.095 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:53.092+0000 7faa9d99c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:53.097 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:41:53.095+0000 7faa9d99c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:41:53.579 INFO:tasks.rados.rados.0.trial169.stdout:137: done (4 left) 2026-02-25T08:41:53.579 INFO:tasks.rados.rados.0.trial169.stdout:141: done (3 left) 2026-02-25T08:41:53.579 INFO:tasks.rados.rados.0.trial169.stdout:142: snap_create 2026-02-25T08:41:54.583 INFO:tasks.rados.rados.0.trial169.stdout:142: done (3 left) 2026-02-25T08:41:54.583 INFO:tasks.rados.rados.0.trial169.stdout:143: read oid 36 snap -1 2026-02-25T08:41:54.584 INFO:tasks.rados.rados.0.trial169.stdout:143: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:41:54.584 INFO:tasks.rados.rados.0.trial169.stdout:144: snap_remove snap 10 2026-02-25T08:41:55.584 INFO:tasks.rados.rados.0.trial169.stdout:144: done (4 left) 2026-02-25T08:41:55.584 INFO:tasks.rados.rados.0.trial169.stdout:145: write (excl) oid 1 current snap is 11 2026-02-25T08:41:55.584 INFO:tasks.rados.rados.0.trial169.stdout:145: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2026-02-25T08:41:55.588 INFO:tasks.rados.rados.0.trial169.stdout:145: writing trial16941936-1 from 729437 to 1256979 tid 1 2026-02-25T08:41:55.593 INFO:tasks.rados.rados.0.trial169.stdout:145: writing trial16941936-1 from 1925626 to 2568977 tid 2 2026-02-25T08:41:55.597 INFO:tasks.rados.rados.0.trial169.stdout:145: writing trial16941936-1 from 3266551 to 3529437 tid 3 2026-02-25T08:41:55.597 INFO:tasks.rados.rados.0.trial169.stdout:146: read oid 42 snap 9 2026-02-25T08:41:55.597 INFO:tasks.rados.rados.0.trial169.stdout:146: expect deleted 2026-02-25T08:41:55.597 INFO:tasks.rados.rados.0.trial169.stdout:147: write (excl) oid 41 current snap is 11 2026-02-25T08:41:55.597 INFO:tasks.rados.rados.0.trial169.stdout:147: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2026-02-25T08:41:55.602 INFO:tasks.rados.rados.0.trial169.stdout:147: writing trial16941936-41 from 746244 to 1465388 tid 1 2026-02-25T08:41:55.606 INFO:tasks.rados.rados.0.trial169.stdout:147: writing trial16941936-41 from 2100461 to 2603823 tid 2 2026-02-25T08:41:55.610 INFO:tasks.rados.rados.0.trial169.stdout:147: writing trial16941936-41 from 3126680 to 3546244 tid 3 2026-02-25T08:41:55.611 INFO:tasks.rados.rados.0.trial169.stdout:148: read oid 44 snap 9 2026-02-25T08:41:55.611 INFO:tasks.rados.rados.0.trial169.stdout:148: expect (ObjNum 54 snap 1 seq_num 54) 2026-02-25T08:41:55.611 INFO:tasks.rados.rados.0.trial169.stdout:149: read oid 23 snap -1 2026-02-25T08:41:55.611 INFO:tasks.rados.rados.0.trial169.stdout:149: expect (ObjNum 22 snap 0 seq_num 22) 2026-02-25T08:41:55.611 INFO:tasks.rados.rados.0.trial169.stdout:150: write oid 4 current snap is 11 2026-02-25T08:41:55.611 INFO:tasks.rados.rados.0.trial169.stdout:150: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2026-02-25T08:41:55.616 INFO:tasks.rados.rados.0.trial169.stdout:150: writing trial16941936-4 from 763051 to 1557444 tid 1 2026-02-25T08:41:55.621 INFO:tasks.rados.rados.0.trial169.stdout:150: writing trial16941936-4 from 2158943 to 2805963 tid 2 2026-02-25T08:41:55.624 INFO:tasks.rados.rados.0.trial169.stdout:150: writing trial16941936-4 from 3437750 to 3563051 tid 3 2026-02-25T08:41:55.624 INFO:tasks.rados.rados.0.trial169.stdout:151: delete oid 11 current snap is 11 2026-02-25T08:41:55.636 INFO:tasks.rados.rados.0.trial169.stdout:147: finishing write tid 1 to trial16941936-41 2026-02-25T08:41:55.650 INFO:tasks.rados.rados.0.trial169.stdout:147: finishing write tid 2 to trial16941936-41 2026-02-25T08:41:55.654 INFO:tasks.rados.rados.0.trial169.stdout:147: finishing write tid 3 to trial16941936-41 2026-02-25T08:41:55.654 INFO:tasks.rados.rados.0.trial169.stdout:150: finishing write tid 1 to trial16941936-4 2026-02-25T08:41:55.657 INFO:tasks.rados.rados.0.trial169.stdout:147: finishing write tid 4 to trial16941936-41 2026-02-25T08:41:55.659 INFO:tasks.rados.rados.0.trial169.stdout:150: finishing write tid 2 to trial16941936-4 2026-02-25T08:41:55.659 INFO:tasks.rados.rados.0.trial169.stdout:147: finishing write tid 5 to trial16941936-41 2026-02-25T08:41:55.659 INFO:tasks.rados.rados.0.trial169.stdout:147: oid 41 updating version 0 to 41 2026-02-25T08:41:55.659 INFO:tasks.rados.rados.0.trial169.stdout:147: oid 41 version 41 is already newer than 41 2026-02-25T08:41:55.660 INFO:tasks.rados.rados.0.trial169.stdout:147: oid 41 version 41 is already newer than 41 2026-02-25T08:41:55.660 INFO:tasks.rados.rados.0.trial169.stdout:147: oid 41 version 41 is already newer than 38 2026-02-25T08:41:55.660 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 41 v 41 (ObjNum 92 snap 11 seq_num 92) dirty exists 2026-02-25T08:41:55.660 INFO:tasks.rados.rados.0.trial169.stdout:147: left oid 41 (ObjNum 92 snap 11 seq_num 92) 2026-02-25T08:41:55.663 INFO:tasks.rados.rados.0.trial169.stdout:150: finishing write tid 3 to trial16941936-4 2026-02-25T08:41:55.666 INFO:tasks.rados.rados.0.trial169.stdout:150: finishing write tid 4 to trial16941936-4 2026-02-25T08:41:55.667 INFO:tasks.rados.rados.0.trial169.stdout:150: finishing write tid 5 to trial16941936-4 2026-02-25T08:41:55.667 INFO:tasks.rados.rados.0.trial169.stdout:150: oid 4 updating version 0 to 54 2026-02-25T08:41:55.667 INFO:tasks.rados.rados.0.trial169.stdout:150: oid 4 version 54 is already newer than 54 2026-02-25T08:41:55.667 INFO:tasks.rados.rados.0.trial169.stdout:150: oid 4 version 54 is already newer than 54 2026-02-25T08:41:55.667 INFO:tasks.rados.rados.0.trial169.stdout:150: oid 4 version 54 is already newer than 54 2026-02-25T08:41:55.667 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 4 v 54 (ObjNum 93 snap 11 seq_num 93) dirty exists 2026-02-25T08:41:55.667 INFO:tasks.rados.rados.0.trial169.stdout:150: left oid 4 (ObjNum 93 snap 11 seq_num 93) 2026-02-25T08:41:57.261 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3] 2026-02-25T08:41:57.261 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2026-02-25T08:41:57.262 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2026-02-25T08:41:57.262 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-25T08:41:57.262 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log 2026-02-25T08:41:57.265 INFO:tasks.ceph.osd.0:Started 2026-02-25T08:41:57.265 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:41:57.335 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:41:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:41:57.338 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:42:02.340 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:42:02.416 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:02.419 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:02.419 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:42:06.020 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2026-02-25T08:42:07.421 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:42:07.493 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:07.496 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:07.497 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:42:11.524 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~6s 2026-02-25T08:42:12.498 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:42:12.571 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:12.574 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:12.575 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:42:17.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~11s 2026-02-25T08:42:17.576 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:42:17.648 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:17.650 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:17.651 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:42:22.130 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:22.215 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:22.216 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:22.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~17s 2026-02-25T08:42:22.651 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:42:22.723 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:22.726 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:42:23.170 INFO:tasks.rados.rados.0.trial169.stdout:140: finishing write tid 1 to trial16941936-27 2026-02-25T08:42:23.171 INFO:tasks.rados.rados.0.trial169.stdout:140: finishing write tid 2 to trial16941936-27 2026-02-25T08:42:23.171 INFO:tasks.rados.rados.0.trial169.stdout:140: finishing write tid 3 to trial16941936-27 2026-02-25T08:42:23.172 INFO:tasks.rados.rados.0.trial169.stdout:140: finishing write tid 4 to trial16941936-27 2026-02-25T08:42:23.176 INFO:tasks.rados.rados.0.trial169.stdout:140: finishing write tid 5 to trial16941936-27 2026-02-25T08:42:23.176 INFO:tasks.rados.rados.0.trial169.stdout:140: oid 27 updating version 0 to 16 2026-02-25T08:42:23.176 INFO:tasks.rados.rados.0.trial169.stdout:140: oid 27 version 16 is already newer than 14 2026-02-25T08:42:23.177 INFO:tasks.rados.rados.0.trial169.stdout:140: oid 27 updating version 16 to 17 2026-02-25T08:42:23.177 INFO:tasks.rados.rados.0.trial169.stdout:140: oid 27 version 17 is already newer than 15 2026-02-25T08:42:23.177 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 27 v 17 (ObjNum 90 snap 9 seq_num 90) dirty exists 2026-02-25T08:42:23.177 INFO:tasks.rados.rados.0.trial169.stdout:140: left oid 27 (ObjNum 90 snap 9 seq_num 90) 2026-02-25T08:42:23.180 INFO:tasks.rados.rados.0.trial169.stdout:140: done (10 left) 2026-02-25T08:42:23.180 INFO:tasks.rados.rados.0.trial169.stdout:146: done (9 left) 2026-02-25T08:42:23.180 INFO:tasks.rados.rados.0.trial169.stdout:147: done (8 left) 2026-02-25T08:42:23.180 INFO:tasks.rados.rados.0.trial169.stdout:150: done (7 left) 2026-02-25T08:42:23.181 INFO:tasks.rados.rados.0.trial169.stdout:151: done (6 left) 2026-02-25T08:42:23.181 INFO:tasks.rados.rados.0.trial169.stdout:152: delete oid 42 current snap is 11 2026-02-25T08:42:23.187 INFO:tasks.rados.rados.0.trial169.stdout:149: expect (ObjNum 22 snap 0 seq_num 22) 2026-02-25T08:42:23.237 INFO:tasks.rados.rados.0.trial169.stdout:145: finishing write tid 1 to trial16941936-1 2026-02-25T08:42:23.237 INFO:tasks.rados.rados.0.trial169.stdout:136: finishing write tid 1 to trial16941936-20 2026-02-25T08:42:23.237 INFO:tasks.rados.rados.0.trial169.stdout:145: finishing write tid 2 to trial16941936-1 2026-02-25T08:42:23.237 INFO:tasks.rados.rados.0.trial169.stdout:145: finishing write tid 3 to trial16941936-1 2026-02-25T08:42:23.237 INFO:tasks.rados.rados.0.trial169.stdout:145: finishing write tid 4 to trial16941936-1 2026-02-25T08:42:23.237 INFO:tasks.rados.rados.0.trial169.stdout:149: done (6 left) 2026-02-25T08:42:23.237 INFO:tasks.rados.rados.0.trial169.stdout:152: done (5 left) 2026-02-25T08:42:23.238 INFO:tasks.rados.rados.0.trial169.stdout:153: write (excl) oid 11 current snap is 11 2026-02-25T08:42:23.238 INFO:tasks.rados.rados.0.trial169.stdout:153: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2026-02-25T08:42:23.240 INFO:tasks.rados.rados.0.trial169.stdout:153: writing trial16941936-11 from 413472 to 1033612 tid 1 2026-02-25T08:42:23.245 INFO:tasks.rados.rados.0.trial169.stdout:153: writing trial16941936-11 from 1818036 to 2328736 tid 2 2026-02-25T08:42:23.251 INFO:tasks.rados.rados.0.trial169.stdout:153: writing trial16941936-11 from 3120019 to 3613472 tid 3 2026-02-25T08:42:23.251 INFO:tasks.rados.rados.0.trial169.stdout:154: read oid 19 snap -1 2026-02-25T08:42:23.252 INFO:tasks.rados.rados.0.trial169.stdout:154: expect deleted 2026-02-25T08:42:23.252 INFO:tasks.rados.rados.0.trial169.stdout:155: read oid 45 snap -1 2026-02-25T08:42:23.252 INFO:tasks.rados.rados.0.trial169.stdout:155: expect (ObjNum 53 snap 1 seq_num 53) 2026-02-25T08:42:23.252 INFO:tasks.rados.rados.0.trial169.stdout:156: write (excl) oid 10 current snap is 11 2026-02-25T08:42:23.252 INFO:tasks.rados.rados.0.trial169.stdout:145: finishing write tid 5 to trial16941936-1 2026-02-25T08:42:23.252 INFO:tasks.rados.rados.0.trial169.stdout:145: oid 1 updating version 0 to 22 2026-02-25T08:42:23.252 INFO:tasks.rados.rados.0.trial169.stdout:145: oid 1 version 22 is already newer than 19 2026-02-25T08:42:23.252 INFO:tasks.rados.rados.0.trial169.stdout:145: oid 1 version 22 is already newer than 22 2026-02-25T08:42:23.252 INFO:tasks.rados.rados.0.trial169.stdout:145: oid 1 version 22 is already newer than 22 2026-02-25T08:42:23.252 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 1 v 22 (ObjNum 91 snap 11 seq_num 91) dirty exists 2026-02-25T08:42:23.252 INFO:tasks.rados.rados.0.trial169.stdout:145: left oid 1 (ObjNum 91 snap 11 seq_num 91) 2026-02-25T08:42:23.253 INFO:tasks.rados.rados.0.trial169.stdout:136: finishing write tid 2 to trial16941936-20 2026-02-25T08:42:23.253 INFO:tasks.rados.rados.0.trial169.stdout:156: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2026-02-25T08:42:23.255 INFO:tasks.rados.rados.0.trial169.stdout:156: writing trial16941936-10 from 430279 to 1125668 tid 1 2026-02-25T08:42:23.260 INFO:tasks.rados.rados.0.trial169.stdout:156: writing trial16941936-10 from 1876518 to 2530876 tid 2 2026-02-25T08:42:23.265 INFO:tasks.rados.rados.0.trial169.stdout:156: writing trial16941936-10 from 3031089 to 3567886 tid 3 2026-02-25T08:42:23.265 INFO:tasks.rados.rados.0.trial169.stdout:145: done (8 left) 2026-02-25T08:42:23.265 INFO:tasks.rados.rados.0.trial169.stdout:157: delete oid 13 current snap is 11 2026-02-25T08:42:23.265 INFO:tasks.rados.rados.0.trial169.stdout:136: finishing write tid 3 to trial16941936-20 2026-02-25T08:42:23.265 INFO:tasks.rados.rados.0.trial169.stdout:136: finishing write tid 4 to trial16941936-20 2026-02-25T08:42:23.265 INFO:tasks.rados.rados.0.trial169.stdout:136: finishing write tid 5 to trial16941936-20 2026-02-25T08:42:23.265 INFO:tasks.rados.rados.0.trial169.stdout:136: oid 20 updating version 0 to 24 2026-02-25T08:42:23.265 INFO:tasks.rados.rados.0.trial169.stdout:136: oid 20 updating version 24 to 27 2026-02-25T08:42:23.265 INFO:tasks.rados.rados.0.trial169.stdout:136: oid 20 version 27 is already newer than 27 2026-02-25T08:42:23.265 INFO:tasks.rados.rados.0.trial169.stdout:136: oid 20 version 27 is already newer than 27 2026-02-25T08:42:23.266 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 20 v 27 (ObjNum 86 snap 9 seq_num 86) dirty exists 2026-02-25T08:42:23.266 INFO:tasks.rados.rados.0.trial169.stdout:136: left oid 20 (ObjNum 86 snap 9 seq_num 86) 2026-02-25T08:42:23.266 INFO:tasks.rados.rados.0.trial169.stdout:139: finishing write tid 1 to trial16941936-30 2026-02-25T08:42:23.266 INFO:tasks.rados.rados.0.trial169.stdout:153: finishing write tid 1 to trial16941936-11 2026-02-25T08:42:23.266 INFO:tasks.rados.rados.0.trial169.stdout:155: expect (ObjNum 53 snap 1 seq_num 53) 2026-02-25T08:42:23.314 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2303714 2026-02-25T08:42:23.337 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2667694 2026-02-25T08:42:23.345 INFO:tasks.rados.rados.0.trial169.stdout:136: done (8 left) 2026-02-25T08:42:23.345 INFO:tasks.rados.rados.0.trial169.stdout:154: done (7 left) 2026-02-25T08:42:23.345 INFO:tasks.rados.rados.0.trial169.stdout:155: done (6 left) 2026-02-25T08:42:23.345 INFO:tasks.rados.rados.0.trial169.stdout:157: done (5 left) 2026-02-25T08:42:23.345 INFO:tasks.rados.rados.0.trial169.stdout:158: read oid 14 snap -1 2026-02-25T08:42:23.345 INFO:tasks.rados.rados.0.trial169.stdout:158: expect (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:42:23.345 INFO:tasks.rados.rados.0.trial169.stdout:159: write oid 24 current snap is 11 2026-02-25T08:42:23.345 INFO:tasks.rados.rados.0.trial169.stdout:139: finishing write tid 2 to trial16941936-30 2026-02-25T08:42:23.346 INFO:tasks.rados.rados.0.trial169.stdout:159: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2026-02-25T08:42:23.348 INFO:tasks.rados.rados.0.trial169.stdout:159: writing trial16941936-24 from 463893 to 1026133 tid 1 2026-02-25T08:42:23.353 INFO:tasks.rados.rados.0.trial169.stdout:159: writing trial16941936-24 from 1593482 to 2251509 tid 2 2026-02-25T08:42:23.358 INFO:tasks.rados.rados.0.trial169.stdout:159: writing trial16941936-24 from 2685935 to 3245276 tid 3 2026-02-25T08:42:23.358 INFO:tasks.rados.rados.0.trial169.stdout:160: delete oid 31 current snap is 11 2026-02-25T08:42:23.358 INFO:tasks.rados.rados.0.trial169.stdout:153: finishing write tid 2 to trial16941936-11 2026-02-25T08:42:23.358 INFO:tasks.rados.rados.0.trial169.stdout:153: finishing write tid 3 to trial16941936-11 2026-02-25T08:42:23.358 INFO:tasks.rados.rados.0.trial169.stdout:153: finishing write tid 4 to trial16941936-11 2026-02-25T08:42:23.358 INFO:tasks.rados.rados.0.trial169.stdout:139: finishing write tid 3 to trial16941936-30 2026-02-25T08:42:23.359 INFO:tasks.rados.rados.0.trial169.stdout:153: finishing write tid 5 to trial16941936-11 2026-02-25T08:42:23.359 INFO:tasks.rados.rados.0.trial169.stdout:153: oid 11 updating version 0 to 21 2026-02-25T08:42:23.359 INFO:tasks.rados.rados.0.trial169.stdout:153: oid 11 updating version 21 to 24 2026-02-25T08:42:23.359 INFO:tasks.rados.rados.0.trial169.stdout:153: oid 11 version 24 is already newer than 24 2026-02-25T08:42:23.359 INFO:tasks.rados.rados.0.trial169.stdout:153: oid 11 version 24 is already newer than 24 2026-02-25T08:42:23.359 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 11 v 24 (ObjNum 96 snap 11 seq_num 96) dirty exists 2026-02-25T08:42:23.359 INFO:tasks.rados.rados.0.trial169.stdout:153: left oid 11 (ObjNum 96 snap 11 seq_num 96) 2026-02-25T08:42:23.359 INFO:tasks.rados.rados.0.trial169.stdout:139: finishing write tid 4 to trial16941936-30 2026-02-25T08:42:23.359 INFO:tasks.rados.rados.0.trial169.stdout:139: finishing write tid 5 to trial16941936-30 2026-02-25T08:42:23.359 INFO:tasks.rados.rados.0.trial169.stdout:139: oid 30 updating version 0 to 31 2026-02-25T08:42:23.359 INFO:tasks.rados.rados.0.trial169.stdout:139: oid 30 version 31 is already newer than 31 2026-02-25T08:42:23.359 INFO:tasks.rados.rados.0.trial169.stdout:139: oid 30 version 31 is already newer than 28 2026-02-25T08:42:23.359 INFO:tasks.rados.rados.0.trial169.stdout:139: oid 30 version 31 is already newer than 31 2026-02-25T08:42:23.360 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 30 v 31 (ObjNum 89 snap 9 seq_num 89) dirty exists 2026-02-25T08:42:23.360 INFO:tasks.rados.rados.0.trial169.stdout:139: left oid 30 (ObjNum 89 snap 9 seq_num 89) 2026-02-25T08:42:23.360 INFO:tasks.rados.rados.0.trial169.stdout:156: finishing write tid 1 to trial16941936-10 2026-02-25T08:42:23.360 INFO:tasks.rados.rados.0.trial169.stdout:143: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:42:23.360 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:23.379 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:23.400 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 309005 2026-02-25T08:42:23.421 INFO:tasks.rados.rados.0.trial169.stdout:156: finishing write tid 2 to trial16941936-10 2026-02-25T08:42:23.421 INFO:tasks.rados.rados.0.trial169.stdout:156: finishing write tid 3 to trial16941936-10 2026-02-25T08:42:23.421 INFO:tasks.rados.rados.0.trial169.stdout:139: done (7 left) 2026-02-25T08:42:23.421 INFO:tasks.rados.rados.0.trial169.stdout:143: done (6 left) 2026-02-25T08:42:23.422 INFO:tasks.rados.rados.0.trial169.stdout:153: done (5 left) 2026-02-25T08:42:23.422 INFO:tasks.rados.rados.0.trial169.stdout:160: done (4 left) 2026-02-25T08:42:23.422 INFO:tasks.rados.rados.0.trial169.stdout:161: read oid 11 snap -1 2026-02-25T08:42:23.422 INFO:tasks.rados.rados.0.trial169.stdout:161: expect (ObjNum 96 snap 11 seq_num 96) 2026-02-25T08:42:23.422 INFO:tasks.rados.rados.0.trial169.stdout:162: write (excl) oid 45 current snap is 11 2026-02-25T08:42:23.422 INFO:tasks.rados.rados.0.trial169.stdout:148: expect (ObjNum 54 snap 1 seq_num 54) 2026-02-25T08:42:23.447 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:23.480 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:23.514 INFO:tasks.rados.rados.0.trial169.stdout:156: finishing write tid 4 to trial16941936-10 2026-02-25T08:42:23.514 INFO:tasks.rados.rados.0.trial169.stdout:156: finishing write tid 5 to trial16941936-10 2026-02-25T08:42:23.514 INFO:tasks.rados.rados.0.trial169.stdout:156: oid 10 updating version 0 to 5 2026-02-25T08:42:23.514 INFO:tasks.rados.rados.0.trial169.stdout:156: oid 10 updating version 5 to 8 2026-02-25T08:42:23.514 INFO:tasks.rados.rados.0.trial169.stdout:156: oid 10 version 8 is already newer than 8 2026-02-25T08:42:23.515 INFO:tasks.rados.rados.0.trial169.stdout:156: oid 10 version 8 is already newer than 8 2026-02-25T08:42:23.515 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 10 v 8 (ObjNum 97 snap 11 seq_num 97) dirty exists 2026-02-25T08:42:23.515 INFO:tasks.rados.rados.0.trial169.stdout:156: left oid 10 (ObjNum 97 snap 11 seq_num 97) 2026-02-25T08:42:23.515 INFO:tasks.rados.rados.0.trial169.stdout:162: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2026-02-25T08:42:23.518 INFO:tasks.rados.rados.0.trial169.stdout:162: writing trial16941936-45 from 497507 to 1210245 tid 1 2026-02-25T08:42:23.522 INFO:tasks.rados.rados.0.trial169.stdout:162: writing trial16941936-45 from 1710446 to 2372142 tid 2 2026-02-25T08:42:23.528 INFO:tasks.rados.rados.0.trial169.stdout:162: writing trial16941936-45 from 3140781 to 3697507 tid 3 2026-02-25T08:42:23.528 INFO:tasks.rados.rados.0.trial169.stdout:148: done (5 left) 2026-02-25T08:42:23.528 INFO:tasks.rados.rados.0.trial169.stdout:156: done (4 left) 2026-02-25T08:42:23.528 INFO:tasks.rados.rados.0.trial169.stdout:163: read oid 3 snap -1 2026-02-25T08:42:23.528 INFO:tasks.rados.rados.0.trial169.stdout:163: expect deleted 2026-02-25T08:42:23.529 INFO:tasks.rados.rados.0.trial169.stdout:164: write oid 41 current snap is 11 2026-02-25T08:42:23.529 INFO:tasks.rados.rados.0.trial169.stdout:164: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2026-02-25T08:42:23.533 INFO:tasks.rados.rados.0.trial169.stdout:164: writing trial16941936-41 from 514314 to 1302301 tid 1 2026-02-25T08:42:23.536 INFO:tasks.rados.rados.0.trial169.stdout:164: writing trial16941936-41 from 1768928 to 2174282 tid 2 2026-02-25T08:42:23.540 INFO:tasks.rados.rados.0.trial169.stdout:164: writing trial16941936-41 from 2768204 to 3277714 tid 3 2026-02-25T08:42:23.540 INFO:tasks.rados.rados.0.trial169.stdout:165: read oid 48 snap -1 2026-02-25T08:42:23.541 INFO:tasks.rados.rados.0.trial169.stdout:165: expect deleted 2026-02-25T08:42:23.541 INFO:tasks.rados.rados.0.trial169.stdout:166: snap_create 2026-02-25T08:42:23.541 INFO:tasks.rados.rados.0.trial169.stdout:158: expect (ObjNum 51 snap 0 seq_num 51) 2026-02-25T08:42:23.546 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1224155 2026-02-25T08:42:23.599 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:23.632 INFO:tasks.rados.rados.0.trial169.stdout:159: finishing write tid 1 to trial16941936-24 2026-02-25T08:42:23.632 INFO:tasks.rados.rados.0.trial169.stdout:159: finishing write tid 2 to trial16941936-24 2026-02-25T08:42:23.632 INFO:tasks.rados.rados.0.trial169.stdout:159: finishing write tid 3 to trial16941936-24 2026-02-25T08:42:23.632 INFO:tasks.rados.rados.0.trial169.stdout:159: finishing write tid 4 to trial16941936-24 2026-02-25T08:42:23.632 INFO:tasks.rados.rados.0.trial169.stdout:159: finishing write tid 5 to trial16941936-24 2026-02-25T08:42:23.632 INFO:tasks.rados.rados.0.trial169.stdout:159: oid 24 updating version 0 to 22 2026-02-25T08:42:23.632 INFO:tasks.rados.rados.0.trial169.stdout:159: oid 24 version 22 is already newer than 22 2026-02-25T08:42:23.632 INFO:tasks.rados.rados.0.trial169.stdout:159: oid 24 version 22 is already newer than 22 2026-02-25T08:42:23.633 INFO:tasks.rados.rados.0.trial169.stdout:159: oid 24 version 22 is already newer than 22 2026-02-25T08:42:23.633 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 24 v 22 (ObjNum 99 snap 11 seq_num 99) dirty exists 2026-02-25T08:42:23.633 INFO:tasks.rados.rados.0.trial169.stdout:159: left oid 24 (ObjNum 99 snap 11 seq_num 99) 2026-02-25T08:42:23.633 INFO:tasks.rados.rados.0.trial169.stdout:161: expect (ObjNum 96 snap 11 seq_num 96) 2026-02-25T08:42:23.662 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:23.693 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:23.724 INFO:tasks.rados.rados.0.trial169.stdout:162: finishing write tid 1 to trial16941936-45 2026-02-25T08:42:23.724 INFO:tasks.rados.rados.0.trial169.stdout:164: finishing write tid 1 to trial16941936-41 2026-02-25T08:42:23.725 INFO:tasks.rados.rados.0.trial169.stdout:164: finishing write tid 2 to trial16941936-41 2026-02-25T08:42:23.725 INFO:tasks.rados.rados.0.trial169.stdout:162: finishing write tid 2 to trial16941936-45 2026-02-25T08:42:23.725 INFO:tasks.rados.rados.0.trial169.stdout:164: finishing write tid 3 to trial16941936-41 2026-02-25T08:42:23.725 INFO:tasks.rados.rados.0.trial169.stdout:162: finishing write tid 3 to trial16941936-45 2026-02-25T08:42:23.725 INFO:tasks.rados.rados.0.trial169.stdout:164: finishing write tid 4 to trial16941936-41 2026-02-25T08:42:23.725 INFO:tasks.rados.rados.0.trial169.stdout:164: finishing write tid 5 to trial16941936-41 2026-02-25T08:42:23.725 INFO:tasks.rados.rados.0.trial169.stdout:164: oid 41 updating version 0 to 48 2026-02-25T08:42:23.725 INFO:tasks.rados.rados.0.trial169.stdout:164: oid 41 version 48 is already newer than 48 2026-02-25T08:42:23.725 INFO:tasks.rados.rados.0.trial169.stdout:164: oid 41 version 48 is already newer than 48 2026-02-25T08:42:23.725 INFO:tasks.rados.rados.0.trial169.stdout:164: oid 41 version 48 is already newer than 48 2026-02-25T08:42:23.725 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 41 v 48 (ObjNum 102 snap 11 seq_num 102) dirty exists 2026-02-25T08:42:23.726 INFO:tasks.rados.rados.0.trial169.stdout:164: left oid 41 (ObjNum 102 snap 11 seq_num 102) 2026-02-25T08:42:23.726 INFO:tasks.rados.rados.0.trial169.stdout:162: finishing write tid 4 to trial16941936-45 2026-02-25T08:42:23.726 INFO:tasks.rados.rados.0.trial169.stdout:162: finishing write tid 5 to trial16941936-45 2026-02-25T08:42:23.726 INFO:tasks.rados.rados.0.trial169.stdout:162: oid 45 updating version 0 to 48 2026-02-25T08:42:23.726 INFO:tasks.rados.rados.0.trial169.stdout:162: oid 45 version 48 is already newer than 48 2026-02-25T08:42:23.726 INFO:tasks.rados.rados.0.trial169.stdout:162: oid 45 version 48 is already newer than 45 2026-02-25T08:42:23.726 INFO:tasks.rados.rados.0.trial169.stdout:162: oid 45 version 48 is already newer than 48 2026-02-25T08:42:23.726 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 45 v 48 (ObjNum 101 snap 11 seq_num 101) dirty exists 2026-02-25T08:42:23.726 INFO:tasks.rados.rados.0.trial169.stdout:162: left oid 45 (ObjNum 101 snap 11 seq_num 101) 2026-02-25T08:42:24.135 INFO:tasks.rados.rados.0.trial169.stdout:158: done (7 left) 2026-02-25T08:42:24.135 INFO:tasks.rados.rados.0.trial169.stdout:159: done (6 left) 2026-02-25T08:42:24.136 INFO:tasks.rados.rados.0.trial169.stdout:161: done (5 left) 2026-02-25T08:42:24.136 INFO:tasks.rados.rados.0.trial169.stdout:162: done (4 left) 2026-02-25T08:42:24.136 INFO:tasks.rados.rados.0.trial169.stdout:163: done (3 left) 2026-02-25T08:42:24.136 INFO:tasks.rados.rados.0.trial169.stdout:164: done (2 left) 2026-02-25T08:42:24.136 INFO:tasks.rados.rados.0.trial169.stdout:165: done (1 left) 2026-02-25T08:42:24.136 INFO:tasks.rados.rados.0.trial169.stdout:166: done (0 left) 2026-02-25T08:42:24.136 INFO:tasks.rados.rados.0.trial169.stdout:167: write oid 20 current snap is 12 2026-02-25T08:42:24.136 INFO:tasks.rados.rados.0.trial169.stdout:167: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2026-02-25T08:42:24.139 INFO:tasks.rados.rados.0.trial169.stdout:167: writing trial16941936-20 from 531121 to 994357 tid 1 2026-02-25T08:42:24.145 INFO:tasks.rados.rados.0.trial169.stdout:167: writing trial16941936-20 from 1711057 to 2376422 tid 2 2026-02-25T08:42:24.151 INFO:tasks.rados.rados.0.trial169.stdout:167: writing trial16941936-20 from 3079274 to 3731121 tid 3 2026-02-25T08:42:24.151 INFO:tasks.rados.rados.0.trial169.stdout:168: read oid 25 snap -1 2026-02-25T08:42:24.151 INFO:tasks.rados.rados.0.trial169.stdout:168: expect (ObjNum 78 snap 8 seq_num 78) 2026-02-25T08:42:24.151 INFO:tasks.rados.rados.0.trial169.stdout:169: read oid 36 snap -1 2026-02-25T08:42:24.151 INFO:tasks.rados.rados.0.trial169.stdout:169: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:42:24.151 INFO:tasks.rados.rados.0.trial169.stdout:170: snap_remove snap 9 2026-02-25T08:42:24.217 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:24.296 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:24.293+0000 7f857d70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:24.299 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:24.297+0000 7f857d70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:24.302 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:24.342 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:24.343 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:25.137 INFO:tasks.rados.rados.0.trial169.stdout:170: done (3 left) 2026-02-25T08:42:25.137 INFO:tasks.rados.rados.0.trial169.stdout:171: read oid 26 snap -1 2026-02-25T08:42:25.138 INFO:tasks.rados.rados.0.trial169.stdout:171: expect (ObjNum 75 snap 8 seq_num 75) 2026-02-25T08:42:25.138 INFO:tasks.rados.rados.0.trial169.stdout:172: write (excl) oid 2 current snap is 12 2026-02-25T08:42:25.138 INFO:tasks.rados.rados.0.trial169.stdout:172: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2026-02-25T08:42:25.141 INFO:tasks.rados.rados.0.trial169.stdout:172: writing trial16941936-2 from 547928 to 1086413 tid 1 2026-02-25T08:42:25.145 INFO:tasks.rados.rados.0.trial169.stdout:172: writing trial16941936-2 from 1769539 to 2178562 tid 2 2026-02-25T08:42:25.149 INFO:tasks.rados.rados.0.trial169.stdout:172: writing trial16941936-2 from 2706697 to 3238751 tid 3 2026-02-25T08:42:25.149 INFO:tasks.rados.rados.0.trial169.stdout:173: snap_create 2026-02-25T08:42:25.149 INFO:tasks.rados.rados.0.trial169.stdout:168: expect (ObjNum 78 snap 8 seq_num 78) 2026-02-25T08:42:25.189 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:25.228 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 296641 2026-02-25T08:42:25.265 INFO:tasks.rados.rados.0.trial169.stdout:169: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:42:25.279 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1438850 2026-02-25T08:42:25.295 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 943289 2026-02-25T08:42:25.310 INFO:tasks.rados.rados.0.trial169.stdout:167: finishing write tid 1 to trial16941936-20 2026-02-25T08:42:25.310 INFO:tasks.rados.rados.0.trial169.stdout:167: finishing write tid 2 to trial16941936-20 2026-02-25T08:42:25.310 INFO:tasks.rados.rados.0.trial169.stdout:167: finishing write tid 3 to trial16941936-20 2026-02-25T08:42:25.310 INFO:tasks.rados.rados.0.trial169.stdout:167: finishing write tid 4 to trial16941936-20 2026-02-25T08:42:25.310 INFO:tasks.rados.rados.0.trial169.stdout:167: finishing write tid 5 to trial16941936-20 2026-02-25T08:42:25.310 INFO:tasks.rados.rados.0.trial169.stdout:167: oid 20 updating version 0 to 36 2026-02-25T08:42:25.310 INFO:tasks.rados.rados.0.trial169.stdout:167: oid 20 version 36 is already newer than 36 2026-02-25T08:42:25.310 INFO:tasks.rados.rados.0.trial169.stdout:167: oid 20 version 36 is already newer than 36 2026-02-25T08:42:25.310 INFO:tasks.rados.rados.0.trial169.stdout:167: oid 20 version 36 is already newer than 36 2026-02-25T08:42:25.310 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 20 v 36 (ObjNum 103 snap 12 seq_num 103) dirty exists 2026-02-25T08:42:25.311 INFO:tasks.rados.rados.0.trial169.stdout:167: left oid 20 (ObjNum 103 snap 12 seq_num 103) 2026-02-25T08:42:25.311 INFO:tasks.rados.rados.0.trial169.stdout:172: finishing write tid 1 to trial16941936-2 2026-02-25T08:42:25.311 INFO:tasks.rados.rados.0.trial169.stdout:171: expect (ObjNum 75 snap 8 seq_num 75) 2026-02-25T08:42:25.366 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 656582 2026-02-25T08:42:25.396 INFO:tasks.rados.rados.0.trial169.stdout:172: finishing write tid 2 to trial16941936-2 2026-02-25T08:42:25.396 INFO:tasks.rados.rados.0.trial169.stdout:172: finishing write tid 3 to trial16941936-2 2026-02-25T08:42:25.396 INFO:tasks.rados.rados.0.trial169.stdout:172: finishing write tid 4 to trial16941936-2 2026-02-25T08:42:25.396 INFO:tasks.rados.rados.0.trial169.stdout:172: finishing write tid 5 to trial16941936-2 2026-02-25T08:42:25.396 INFO:tasks.rados.rados.0.trial169.stdout:172: oid 2 updating version 0 to 23 2026-02-25T08:42:25.396 INFO:tasks.rados.rados.0.trial169.stdout:172: oid 2 updating version 23 to 26 2026-02-25T08:42:25.397 INFO:tasks.rados.rados.0.trial169.stdout:172: oid 2 version 26 is already newer than 26 2026-02-25T08:42:25.397 INFO:tasks.rados.rados.0.trial169.stdout:172: oid 2 version 26 is already newer than 26 2026-02-25T08:42:25.397 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 2 v 26 (ObjNum 104 snap 12 seq_num 104) dirty exists 2026-02-25T08:42:25.397 INFO:tasks.rados.rados.0.trial169.stdout:172: left oid 2 (ObjNum 104 snap 12 seq_num 104) 2026-02-25T08:42:26.143 INFO:tasks.rados.rados.0.trial169.stdout:167: done (5 left) 2026-02-25T08:42:26.143 INFO:tasks.rados.rados.0.trial169.stdout:168: done (4 left) 2026-02-25T08:42:26.143 INFO:tasks.rados.rados.0.trial169.stdout:169: done (3 left) 2026-02-25T08:42:26.143 INFO:tasks.rados.rados.0.trial169.stdout:171: done (2 left) 2026-02-25T08:42:26.143 INFO:tasks.rados.rados.0.trial169.stdout:172: done (1 left) 2026-02-25T08:42:26.143 INFO:tasks.rados.rados.0.trial169.stdout:173: done (0 left) 2026-02-25T08:42:26.143 INFO:tasks.rados.rados.0.trial169.stdout:174: read oid 49 snap -1 2026-02-25T08:42:26.144 INFO:tasks.rados.rados.0.trial169.stdout:174: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:42:26.144 INFO:tasks.rados.rados.0.trial169.stdout:175: read oid 29 snap -1 2026-02-25T08:42:26.144 INFO:tasks.rados.rados.0.trial169.stdout:175: expect (ObjNum 28 snap 0 seq_num 28) 2026-02-25T08:42:26.144 INFO:tasks.rados.rados.0.trial169.stdout:176: snap_remove snap 12 2026-02-25T08:42:26.344 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:26.422 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:26.419+0000 7f6580f5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:26.425 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:26.423+0000 7f6580f5b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:26.429 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:26.468 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:26.469 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:27.146 INFO:tasks.rados.rados.0.trial169.stdout:176: done (2 left) 2026-02-25T08:42:27.146 INFO:tasks.rados.rados.0.trial169.stdout:177: write (excl) oid 7 current snap is 13 2026-02-25T08:42:27.146 INFO:tasks.rados.rados.0.trial169.stdout:177: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2026-02-25T08:42:27.150 INFO:tasks.rados.rados.0.trial169.stdout:177: writing trial16941936-7 from 564735 to 1178469 tid 1 2026-02-25T08:42:27.156 INFO:tasks.rados.rados.0.trial169.stdout:177: writing trial16941936-7 from 1828021 to 2497055 tid 2 2026-02-25T08:42:27.161 INFO:tasks.rados.rados.0.trial169.stdout:177: writing trial16941936-7 from 3134120 to 3764735 tid 3 2026-02-25T08:42:27.161 INFO:tasks.rados.rados.0.trial169.stdout:178: delete oid 2 current snap is 13 2026-02-25T08:42:27.161 INFO:tasks.rados.rados.0.trial169.stdout:175: expect (ObjNum 28 snap 0 seq_num 28) 2026-02-25T08:42:27.172 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1112819 2026-02-25T08:42:27.186 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:27.207 INFO:tasks.rados.rados.0.trial169.stdout:174: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:42:27.207 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:27.234 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1078745 2026-02-25T08:42:27.273 INFO:tasks.rados.rados.0.trial169.stdout:177: finishing write tid 1 to trial16941936-7 2026-02-25T08:42:27.274 INFO:tasks.rados.rados.0.trial169.stdout:177: finishing write tid 2 to trial16941936-7 2026-02-25T08:42:27.274 INFO:tasks.rados.rados.0.trial169.stdout:177: finishing write tid 3 to trial16941936-7 2026-02-25T08:42:27.274 INFO:tasks.rados.rados.0.trial169.stdout:177: finishing write tid 4 to trial16941936-7 2026-02-25T08:42:27.274 INFO:tasks.rados.rados.0.trial169.stdout:177: finishing write tid 5 to trial16941936-7 2026-02-25T08:42:27.274 INFO:tasks.rados.rados.0.trial169.stdout:177: oid 7 updating version 0 to 61 2026-02-25T08:42:27.274 INFO:tasks.rados.rados.0.trial169.stdout:177: oid 7 version 61 is already newer than 58 2026-02-25T08:42:27.274 INFO:tasks.rados.rados.0.trial169.stdout:177: oid 7 version 61 is already newer than 61 2026-02-25T08:42:27.274 INFO:tasks.rados.rados.0.trial169.stdout:177: oid 7 version 61 is already newer than 61 2026-02-25T08:42:27.274 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 7 v 61 (ObjNum 105 snap 13 seq_num 105) dirty exists 2026-02-25T08:42:27.274 INFO:tasks.rados.rados.0.trial169.stdout:177: left oid 7 (ObjNum 105 snap 13 seq_num 105) 2026-02-25T08:42:27.274 INFO:tasks.rados.rados.0.trial169.stdout:174: done (3 left) 2026-02-25T08:42:27.275 INFO:tasks.rados.rados.0.trial169.stdout:175: done (2 left) 2026-02-25T08:42:27.275 INFO:tasks.rados.rados.0.trial169.stdout:177: done (1 left) 2026-02-25T08:42:27.275 INFO:tasks.rados.rados.0.trial169.stdout:178: done (0 left) 2026-02-25T08:42:27.275 INFO:tasks.rados.rados.0.trial169.stdout:179: snap_remove snap 11 2026-02-25T08:42:27.727 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:42:27.798 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:27.801 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:27.801 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:42:28.036 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~22s 2026-02-25T08:42:28.148 INFO:tasks.rados.rados.0.trial169.stdout:179: done (0 left) 2026-02-25T08:42:28.148 INFO:tasks.rados.rados.0.trial169.stdout:180: delete oid 14 current snap is 13 2026-02-25T08:42:28.176 INFO:tasks.rados.rados.0.trial169.stdout:180: done (0 left) 2026-02-25T08:42:28.177 INFO:tasks.rados.rados.0.trial169.stdout:181: write (excl) oid 13 current snap is 13 2026-02-25T08:42:28.177 INFO:tasks.rados.rados.0.trial169.stdout:181: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2026-02-25T08:42:28.181 INFO:tasks.rados.rados.0.trial169.stdout:181: writing trial16941936-13 from 615156 to 1170990 tid 1 2026-02-25T08:42:28.184 INFO:tasks.rados.rados.0.trial169.stdout:181: writing trial16941936-13 from 1603467 to 2019828 tid 2 2026-02-25T08:42:28.190 INFO:tasks.rados.rados.0.trial169.stdout:181: writing trial16941936-13 from 2816389 to 3509884 tid 3 2026-02-25T08:42:28.190 INFO:tasks.rados.rados.0.trial169.stdout:182: write oid 29 current snap is 13 2026-02-25T08:42:28.190 INFO:tasks.rados.rados.0.trial169.stdout:182: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2026-02-25T08:42:28.194 INFO:tasks.rados.rados.0.trial169.stdout:182: writing trial16941936-29 from 631963 to 1263046 tid 1 2026-02-25T08:42:28.199 INFO:tasks.rados.rados.0.trial169.stdout:182: writing trial16941936-29 from 2061949 to 2621968 tid 2 2026-02-25T08:42:28.204 INFO:tasks.rados.rados.0.trial169.stdout:182: writing trial16941936-29 from 3243812 to 3748579 tid 3 2026-02-25T08:42:28.204 INFO:tasks.rados.rados.0.trial169.stdout:183: snap_create 2026-02-25T08:42:28.204 INFO:tasks.rados.rados.0.trial169.stdout:181: finishing write tid 1 to trial16941936-13 2026-02-25T08:42:28.217 INFO:tasks.rados.rados.0.trial169.stdout:181: finishing write tid 2 to trial16941936-13 2026-02-25T08:42:28.219 INFO:tasks.rados.rados.0.trial169.stdout:182: finishing write tid 1 to trial16941936-29 2026-02-25T08:42:28.222 INFO:tasks.rados.rados.0.trial169.stdout:181: finishing write tid 3 to trial16941936-13 2026-02-25T08:42:28.225 INFO:tasks.rados.rados.0.trial169.stdout:182: finishing write tid 2 to trial16941936-29 2026-02-25T08:42:28.225 INFO:tasks.rados.rados.0.trial169.stdout:181: finishing write tid 4 to trial16941936-13 2026-02-25T08:42:28.227 INFO:tasks.rados.rados.0.trial169.stdout:181: finishing write tid 5 to trial16941936-13 2026-02-25T08:42:28.227 INFO:tasks.rados.rados.0.trial169.stdout:181: oid 13 updating version 0 to 56 2026-02-25T08:42:28.227 INFO:tasks.rados.rados.0.trial169.stdout:181: oid 13 version 56 is already newer than 56 2026-02-25T08:42:28.227 INFO:tasks.rados.rados.0.trial169.stdout:181: oid 13 version 56 is already newer than 56 2026-02-25T08:42:28.227 INFO:tasks.rados.rados.0.trial169.stdout:181: oid 13 version 56 is already newer than 53 2026-02-25T08:42:28.227 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 13 v 56 (ObjNum 108 snap 13 seq_num 108) dirty exists 2026-02-25T08:42:28.227 INFO:tasks.rados.rados.0.trial169.stdout:181: left oid 13 (ObjNum 108 snap 13 seq_num 108) 2026-02-25T08:42:28.228 INFO:tasks.rados.rados.0.trial169.stdout:182: finishing write tid 3 to trial16941936-29 2026-02-25T08:42:28.231 INFO:tasks.rados.rados.0.trial169.stdout:182: finishing write tid 4 to trial16941936-29 2026-02-25T08:42:28.233 INFO:tasks.rados.rados.0.trial169.stdout:182: finishing write tid 5 to trial16941936-29 2026-02-25T08:42:28.233 INFO:tasks.rados.rados.0.trial169.stdout:182: oid 29 updating version 0 to 67 2026-02-25T08:42:28.233 INFO:tasks.rados.rados.0.trial169.stdout:182: oid 29 version 67 is already newer than 67 2026-02-25T08:42:28.233 INFO:tasks.rados.rados.0.trial169.stdout:182: oid 29 version 67 is already newer than 67 2026-02-25T08:42:28.233 INFO:tasks.rados.rados.0.trial169.stdout:182: oid 29 version 67 is already newer than 67 2026-02-25T08:42:28.233 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 29 v 67 (ObjNum 109 snap 13 seq_num 109) dirty exists 2026-02-25T08:42:28.233 INFO:tasks.rados.rados.0.trial169.stdout:182: left oid 29 (ObjNum 109 snap 13 seq_num 109) 2026-02-25T08:42:28.470 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:28.547 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:28.544+0000 7f77f02cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:28.549 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:28.548+0000 7f77f02cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:28.553 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:28.591 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:29.152 INFO:tasks.rados.rados.0.trial169.stdout:181: done (2 left) 2026-02-25T08:42:29.153 INFO:tasks.rados.rados.0.trial169.stdout:182: done (1 left) 2026-02-25T08:42:29.153 INFO:tasks.rados.rados.0.trial169.stdout:183: done (0 left) 2026-02-25T08:42:29.153 INFO:tasks.rados.rados.0.trial169.stdout:184: write (excl) oid 45 current snap is 14 2026-02-25T08:42:29.153 INFO:tasks.rados.rados.0.trial169.stdout:184: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2026-02-25T08:42:29.157 INFO:tasks.rados.rados.0.trial169.stdout:184: writing trial16941936-45 from 648770 to 1355102 tid 1 2026-02-25T08:42:29.162 INFO:tasks.rados.rados.0.trial169.stdout:184: writing trial16941936-45 from 2120431 to 2540461 tid 2 2026-02-25T08:42:29.167 INFO:tasks.rados.rados.0.trial169.stdout:184: writing trial16941936-45 from 3271235 to 3703627 tid 3 2026-02-25T08:42:29.167 INFO:tasks.rados.rados.0.trial169.stdout:185: snap_remove snap 13 2026-02-25T08:42:30.156 INFO:tasks.rados.rados.0.trial169.stdout:185: done (1 left) 2026-02-25T08:42:30.156 INFO:tasks.rados.rados.0.trial169.stdout:186: write (excl) oid 9 current snap is 14 2026-02-25T08:42:30.156 INFO:tasks.rados.rados.0.trial169.stdout:184: finishing write tid 1 to trial16941936-45 2026-02-25T08:42:30.156 INFO:tasks.rados.rados.0.trial169.stdout:186: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2026-02-25T08:42:30.161 INFO:tasks.rados.rados.0.trial169.stdout:186: writing trial16941936-9 from 665577 to 1447158 tid 1 2026-02-25T08:42:30.165 INFO:tasks.rados.rados.0.trial169.stdout:186: writing trial16941936-9 from 2062560 to 2626248 tid 2 2026-02-25T08:42:30.170 INFO:tasks.rados.rados.0.trial169.stdout:186: writing trial16941936-9 from 3182305 to 3825969 tid 3 2026-02-25T08:42:30.171 INFO:tasks.rados.rados.0.trial169.stdout:187: write (excl) oid 20 current snap is 14 2026-02-25T08:42:30.171 INFO:tasks.rados.rados.0.trial169.stdout:184: finishing write tid 2 to trial16941936-45 2026-02-25T08:42:30.171 INFO:tasks.rados.rados.0.trial169.stdout:184: finishing write tid 3 to trial16941936-45 2026-02-25T08:42:30.171 INFO:tasks.rados.rados.0.trial169.stdout:184: finishing write tid 4 to trial16941936-45 2026-02-25T08:42:30.171 INFO:tasks.rados.rados.0.trial169.stdout:184: finishing write tid 5 to trial16941936-45 2026-02-25T08:42:30.171 INFO:tasks.rados.rados.0.trial169.stdout:184: oid 45 updating version 0 to 54 2026-02-25T08:42:30.171 INFO:tasks.rados.rados.0.trial169.stdout:184: oid 45 version 54 is already newer than 54 2026-02-25T08:42:30.171 INFO:tasks.rados.rados.0.trial169.stdout:184: oid 45 version 54 is already newer than 51 2026-02-25T08:42:30.171 INFO:tasks.rados.rados.0.trial169.stdout:184: oid 45 version 54 is already newer than 54 2026-02-25T08:42:30.171 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 45 v 54 (ObjNum 110 snap 14 seq_num 110) dirty exists 2026-02-25T08:42:30.171 INFO:tasks.rados.rados.0.trial169.stdout:184: left oid 45 (ObjNum 110 snap 14 seq_num 110) 2026-02-25T08:42:30.171 INFO:tasks.rados.rados.0.trial169.stdout:187: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2026-02-25T08:42:30.174 INFO:tasks.rados.rados.0.trial169.stdout:187: writing trial16941936-20 from 682384 to 1139214 tid 1 2026-02-25T08:42:30.178 INFO:tasks.rados.rados.0.trial169.stdout:187: writing trial16941936-20 from 1721042 to 2144741 tid 2 2026-02-25T08:42:30.182 INFO:tasks.rados.rados.0.trial169.stdout:187: writing trial16941936-20 from 2809728 to 3264664 tid 3 2026-02-25T08:42:30.185 INFO:tasks.rados.rados.0.trial169.stdout:187: writing trial16941936-20 from 3731357 to 3882384 tid 4 2026-02-25T08:42:30.185 INFO:tasks.rados.rados.0.trial169.stdout:184: done (2 left) 2026-02-25T08:42:30.185 INFO:tasks.rados.rados.0.trial169.stdout:188: snap_create 2026-02-25T08:42:30.185 INFO:tasks.rados.rados.0.trial169.stdout:186: finishing write tid 1 to trial16941936-9 2026-02-25T08:42:30.194 INFO:tasks.rados.rados.0.trial169.stdout:187: finishing write tid 1 to trial16941936-20 2026-02-25T08:42:30.205 INFO:tasks.rados.rados.0.trial169.stdout:186: finishing write tid 2 to trial16941936-9 2026-02-25T08:42:30.210 INFO:tasks.rados.rados.0.trial169.stdout:186: finishing write tid 3 to trial16941936-9 2026-02-25T08:42:30.210 INFO:tasks.rados.rados.0.trial169.stdout:187: finishing write tid 2 to trial16941936-20 2026-02-25T08:42:30.213 INFO:tasks.rados.rados.0.trial169.stdout:186: finishing write tid 4 to trial16941936-9 2026-02-25T08:42:30.215 INFO:tasks.rados.rados.0.trial169.stdout:186: finishing write tid 5 to trial16941936-9 2026-02-25T08:42:30.215 INFO:tasks.rados.rados.0.trial169.stdout:186: oid 9 updating version 0 to 25 2026-02-25T08:42:30.215 INFO:tasks.rados.rados.0.trial169.stdout:186: oid 9 version 25 is already newer than 22 2026-02-25T08:42:30.215 INFO:tasks.rados.rados.0.trial169.stdout:186: oid 9 version 25 is already newer than 25 2026-02-25T08:42:30.215 INFO:tasks.rados.rados.0.trial169.stdout:186: oid 9 version 25 is already newer than 25 2026-02-25T08:42:30.215 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 9 v 25 (ObjNum 111 snap 14 seq_num 111) dirty exists 2026-02-25T08:42:30.215 INFO:tasks.rados.rados.0.trial169.stdout:186: left oid 9 (ObjNum 111 snap 14 seq_num 111) 2026-02-25T08:42:30.220 INFO:tasks.rados.rados.0.trial169.stdout:187: finishing write tid 3 to trial16941936-20 2026-02-25T08:42:30.225 INFO:tasks.rados.rados.0.trial169.stdout:187: finishing write tid 4 to trial16941936-20 2026-02-25T08:42:30.228 INFO:tasks.rados.rados.0.trial169.stdout:187: finishing write tid 5 to trial16941936-20 2026-02-25T08:42:30.230 INFO:tasks.rados.rados.0.trial169.stdout:187: finishing write tid 6 to trial16941936-20 2026-02-25T08:42:30.230 INFO:tasks.rados.rados.0.trial169.stdout:187: oid 20 updating version 0 to 41 2026-02-25T08:42:30.230 INFO:tasks.rados.rados.0.trial169.stdout:187: oid 20 version 41 is already newer than 37 2026-02-25T08:42:30.230 INFO:tasks.rados.rados.0.trial169.stdout:187: oid 20 version 41 is already newer than 41 2026-02-25T08:42:30.230 INFO:tasks.rados.rados.0.trial169.stdout:187: oid 20 version 41 is already newer than 41 2026-02-25T08:42:30.230 INFO:tasks.rados.rados.0.trial169.stdout:187: oid 20 version 41 is already newer than 41 2026-02-25T08:42:30.230 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 20 v 41 (ObjNum 112 snap 14 seq_num 112) dirty exists 2026-02-25T08:42:30.230 INFO:tasks.rados.rados.0.trial169.stdout:187: left oid 20 (ObjNum 112 snap 14 seq_num 112) 2026-02-25T08:42:30.592 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:30.670 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:30.667+0000 7f52c925a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:30.672 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:30.671+0000 7f52c925a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:30.676 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:30.716 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:30.717 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:31.159 INFO:tasks.rados.rados.0.trial169.stdout:186: done (2 left) 2026-02-25T08:42:31.159 INFO:tasks.rados.rados.0.trial169.stdout:187: done (1 left) 2026-02-25T08:42:31.159 INFO:tasks.rados.rados.0.trial169.stdout:188: done (0 left) 2026-02-25T08:42:31.159 INFO:tasks.rados.rados.0.trial169.stdout:189: snap_create 2026-02-25T08:42:32.162 INFO:tasks.rados.rados.0.trial169.stdout:189: done (0 left) 2026-02-25T08:42:32.162 INFO:tasks.rados.rados.0.trial169.stdout:190: snap_remove snap 15 2026-02-25T08:42:32.718 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:32.793 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:32.791+0000 7f88b36cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:32.796 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:32.794+0000 7f88b36cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:32.799 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:32.802 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:42:32.840 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:32.840 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:32.872 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:32.874 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:32.875 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:42:33.166 INFO:tasks.rados.rados.0.trial169.stdout:190: done (0 left) 2026-02-25T08:42:33.166 INFO:tasks.rados.rados.0.trial169.stdout:191: write (excl) oid 9 current snap is 16 2026-02-25T08:42:33.167 INFO:tasks.rados.rados.0.trial169.stdout:191: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2026-02-25T08:42:33.171 INFO:tasks.rados.rados.0.trial169.stdout:191: writing trial16941936-9 from 699191 to 1231270 tid 1 2026-02-25T08:42:33.175 INFO:tasks.rados.rados.0.trial169.stdout:191: writing trial16941936-9 from 1779524 to 2346881 tid 2 2026-02-25T08:42:33.180 INFO:tasks.rados.rados.0.trial169.stdout:191: writing trial16941936-9 from 2837151 to 3503359 tid 3 2026-02-25T08:42:33.180 INFO:tasks.rados.rados.0.trial169.stdout:192: snap_create 2026-02-25T08:42:33.204 INFO:tasks.rados.rados.0.trial169.stdout:191: finishing write tid 1 to trial16941936-9 2026-02-25T08:42:33.216 INFO:tasks.rados.rados.0.trial169.stdout:191: finishing write tid 2 to trial16941936-9 2026-02-25T08:42:33.222 INFO:tasks.rados.rados.0.trial169.stdout:191: finishing write tid 3 to trial16941936-9 2026-02-25T08:42:33.226 INFO:tasks.rados.rados.0.trial169.stdout:191: finishing write tid 4 to trial16941936-9 2026-02-25T08:42:33.227 INFO:tasks.rados.rados.0.trial169.stdout:191: finishing write tid 5 to trial16941936-9 2026-02-25T08:42:33.228 INFO:tasks.rados.rados.0.trial169.stdout:191: oid 9 updating version 0 to 30 2026-02-25T08:42:33.228 INFO:tasks.rados.rados.0.trial169.stdout:191: oid 9 version 30 is already newer than 30 2026-02-25T08:42:33.228 INFO:tasks.rados.rados.0.trial169.stdout:191: oid 9 version 30 is already newer than 30 2026-02-25T08:42:33.228 INFO:tasks.rados.rados.0.trial169.stdout:191: oid 9 version 30 is already newer than 27 2026-02-25T08:42:33.228 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 9 v 30 (ObjNum 113 snap 16 seq_num 113) dirty exists 2026-02-25T08:42:33.228 INFO:tasks.rados.rados.0.trial169.stdout:191: left oid 9 (ObjNum 113 snap 16 seq_num 113) 2026-02-25T08:42:33.540 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~28s 2026-02-25T08:42:34.172 INFO:tasks.rados.rados.0.trial169.stdout:191: done (1 left) 2026-02-25T08:42:34.173 INFO:tasks.rados.rados.0.trial169.stdout:192: done (0 left) 2026-02-25T08:42:34.173 INFO:tasks.rados.rados.0.trial169.stdout:193: read oid 33 snap -1 2026-02-25T08:42:34.173 INFO:tasks.rados.rados.0.trial169.stdout:193: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-25T08:42:34.173 INFO:tasks.rados.rados.0.trial169.stdout:194: write (excl) oid 31 current snap is 17 2026-02-25T08:42:34.173 INFO:tasks.rados.rados.0.trial169.stdout:194: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2026-02-25T08:42:34.177 INFO:tasks.rados.rados.0.trial169.stdout:194: writing trial16941936-31 from 715998 to 1323326 tid 1 2026-02-25T08:42:34.181 INFO:tasks.rados.rados.0.trial169.stdout:194: writing trial16941936-31 from 1838006 to 2265374 tid 2 2026-02-25T08:42:34.185 INFO:tasks.rados.rados.0.trial169.stdout:194: writing trial16941936-31 from 2864574 to 3342054 tid 3 2026-02-25T08:42:34.187 INFO:tasks.rados.rados.0.trial169.stdout:194: writing trial16941936-31 from 3863835 to 3915998 tid 4 2026-02-25T08:42:34.187 INFO:tasks.rados.rados.0.trial169.stdout:195: delete oid 28 current snap is 17 2026-02-25T08:42:34.206 INFO:tasks.rados.rados.0.trial169.stdout:193: expect (ObjNum 32 snap 0 seq_num 32) 2026-02-25T08:42:34.267 INFO:tasks.rados.rados.0.trial169.stdout:194: finishing write tid 1 to trial16941936-31 2026-02-25T08:42:34.267 INFO:tasks.rados.rados.0.trial169.stdout:194: finishing write tid 2 to trial16941936-31 2026-02-25T08:42:34.267 INFO:tasks.rados.rados.0.trial169.stdout:194: finishing write tid 3 to trial16941936-31 2026-02-25T08:42:34.267 INFO:tasks.rados.rados.0.trial169.stdout:194: finishing write tid 4 to trial16941936-31 2026-02-25T08:42:34.267 INFO:tasks.rados.rados.0.trial169.stdout:194: finishing write tid 5 to trial16941936-31 2026-02-25T08:42:34.267 INFO:tasks.rados.rados.0.trial169.stdout:194: finishing write tid 6 to trial16941936-31 2026-02-25T08:42:34.267 INFO:tasks.rados.rados.0.trial169.stdout:194: oid 31 updating version 0 to 61 2026-02-25T08:42:34.268 INFO:tasks.rados.rados.0.trial169.stdout:194: oid 31 version 61 is already newer than 57 2026-02-25T08:42:34.268 INFO:tasks.rados.rados.0.trial169.stdout:194: oid 31 version 61 is already newer than 61 2026-02-25T08:42:34.268 INFO:tasks.rados.rados.0.trial169.stdout:194: oid 31 version 61 is already newer than 61 2026-02-25T08:42:34.268 INFO:tasks.rados.rados.0.trial169.stdout:194: oid 31 version 61 is already newer than 61 2026-02-25T08:42:34.268 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 31 v 61 (ObjNum 114 snap 17 seq_num 114) dirty exists 2026-02-25T08:42:34.268 INFO:tasks.rados.rados.0.trial169.stdout:194: left oid 31 (ObjNum 114 snap 17 seq_num 114) 2026-02-25T08:42:34.268 INFO:tasks.rados.rados.0.trial169.stdout:193: done (2 left) 2026-02-25T08:42:34.268 INFO:tasks.rados.rados.0.trial169.stdout:194: done (1 left) 2026-02-25T08:42:34.268 INFO:tasks.rados.rados.0.trial169.stdout:195: done (0 left) 2026-02-25T08:42:34.268 INFO:tasks.rados.rados.0.trial169.stdout:196: delete oid 29 current snap is 17 2026-02-25T08:42:34.293 INFO:tasks.rados.rados.0.trial169.stdout:196: done (0 left) 2026-02-25T08:42:34.293 INFO:tasks.rados.rados.0.trial169.stdout:197: read oid 3 snap -1 2026-02-25T08:42:34.293 INFO:tasks.rados.rados.0.trial169.stdout:197: expect deleted 2026-02-25T08:42:34.293 INFO:tasks.rados.rados.0.trial169.stdout:198: write (excl) oid 19 current snap is 17 2026-02-25T08:42:34.293 INFO:tasks.rados.rados.0.trial169.stdout:198: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2026-02-25T08:42:34.297 INFO:tasks.rados.rados.0.trial169.stdout:198: writing trial16941936-19 from 766419 to 1315847 tid 1 2026-02-25T08:42:34.302 INFO:tasks.rados.rados.0.trial169.stdout:198: writing trial16941936-19 from 2013452 to 2588147 tid 2 2026-02-25T08:42:34.307 INFO:tasks.rados.rados.0.trial169.stdout:198: writing trial16941936-19 from 3346843 to 3774492 tid 3 2026-02-25T08:42:34.307 INFO:tasks.rados.rados.0.trial169.stdout:199: snap_remove snap 16 2026-02-25T08:42:34.842 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:34.919 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:34.916+0000 7f7a2d9b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:34.922 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:34.920+0000 7f7a2d9b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:34.925 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:34.965 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:34.966 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:35.176 INFO:tasks.rados.rados.0.trial169.stdout:199: done (2 left) 2026-02-25T08:42:35.176 INFO:tasks.rados.rados.0.trial169.stdout:200: write oid 50 current snap is 17 2026-02-25T08:42:35.176 INFO:tasks.rados.rados.0.trial169.stdout:200: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2026-02-25T08:42:35.182 INFO:tasks.rados.rados.0.trial169.stdout:200: writing trial16941936-50 from 783226 to 1407903 tid 1 2026-02-25T08:42:35.186 INFO:tasks.rados.rados.0.trial169.stdout:200: writing trial16941936-50 from 2071934 to 2506640 tid 2 2026-02-25T08:42:35.190 INFO:tasks.rados.rados.0.trial169.stdout:200: writing trial16941936-50 from 2974266 to 3613187 tid 3 2026-02-25T08:42:35.190 INFO:tasks.rados.rados.0.trial169.stdout:201: write oid 24 current snap is 17 2026-02-25T08:42:35.190 INFO:tasks.rados.rados.0.trial169.stdout:201: seq_num 119 ranges {400033=699926,1614063=385970} 2026-02-25T08:42:35.194 INFO:tasks.rados.rados.0.trial169.stdout:201: writing trial16941936-24 from 400033 to 1099959 tid 1 2026-02-25T08:42:35.198 INFO:tasks.rados.rados.0.trial169.stdout:201: writing trial16941936-24 from 1614063 to 2000033 tid 2 2026-02-25T08:42:35.198 INFO:tasks.rados.rados.0.trial169.stdout:202: write (excl) oid 45 current snap is 17 2026-02-25T08:42:35.198 INFO:tasks.rados.rados.0.trial169.stdout:202: seq_num 120 ranges {416840=775175,1672545=344295} 2026-02-25T08:42:35.202 INFO:tasks.rados.rados.0.trial169.stdout:202: writing trial16941936-45 from 416840 to 1192015 tid 1 2026-02-25T08:42:35.205 INFO:tasks.rados.rados.0.trial169.stdout:202: writing trial16941936-45 from 1672545 to 2016840 tid 2 2026-02-25T08:42:35.206 INFO:tasks.rados.rados.0.trial169.stdout:203: write (excl) oid 37 current snap is 17 2026-02-25T08:42:35.206 INFO:tasks.rados.rados.0.trial169.stdout:203: seq_num 121 ranges {433647=450424,1331027=582033} 2026-02-25T08:42:35.209 INFO:tasks.rados.rados.0.trial169.stdout:203: writing trial16941936-37 from 433647 to 884071 tid 1 2026-02-25T08:42:35.213 INFO:tasks.rados.rados.0.trial169.stdout:203: writing trial16941936-37 from 1331027 to 1913060 tid 2 2026-02-25T08:42:35.213 INFO:tasks.rados.rados.0.trial169.stdout:204: read oid 32 snap -1 2026-02-25T08:42:35.214 INFO:tasks.rados.rados.0.trial169.stdout:204: expect deleted 2026-02-25T08:42:35.214 INFO:tasks.rados.rados.0.trial169.stdout:205: delete oid 27 current snap is 17 2026-02-25T08:42:35.214 INFO:tasks.rados.rados.0.trial169.stdout:198: finishing write tid 1 to trial16941936-19 2026-02-25T08:42:35.214 INFO:tasks.rados.rados.0.trial169.stdout:198: finishing write tid 2 to trial16941936-19 2026-02-25T08:42:35.214 INFO:tasks.rados.rados.0.trial169.stdout:198: finishing write tid 3 to trial16941936-19 2026-02-25T08:42:35.214 INFO:tasks.rados.rados.0.trial169.stdout:198: finishing write tid 4 to trial16941936-19 2026-02-25T08:42:35.214 INFO:tasks.rados.rados.0.trial169.stdout:198: finishing write tid 5 to trial16941936-19 2026-02-25T08:42:35.214 INFO:tasks.rados.rados.0.trial169.stdout:198: oid 19 updating version 0 to 34 2026-02-25T08:42:35.214 INFO:tasks.rados.rados.0.trial169.stdout:198: oid 19 version 34 is already newer than 34 2026-02-25T08:42:35.214 INFO:tasks.rados.rados.0.trial169.stdout:198: oid 19 version 34 is already newer than 31 2026-02-25T08:42:35.214 INFO:tasks.rados.rados.0.trial169.stdout:198: oid 19 version 34 is already newer than 34 2026-02-25T08:42:35.215 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 19 v 34 (ObjNum 117 snap 17 seq_num 117) dirty exists 2026-02-25T08:42:35.215 INFO:tasks.rados.rados.0.trial169.stdout:198: left oid 19 (ObjNum 117 snap 17 seq_num 117) 2026-02-25T08:42:35.233 INFO:tasks.rados.rados.0.trial169.stdout:201: finishing write tid 1 to trial16941936-24 2026-02-25T08:42:35.239 INFO:tasks.rados.rados.0.trial169.stdout:197: done (7 left) 2026-02-25T08:42:35.239 INFO:tasks.rados.rados.0.trial169.stdout:198: done (6 left) 2026-02-25T08:42:35.239 INFO:tasks.rados.rados.0.trial169.stdout:204: done (5 left) 2026-02-25T08:42:35.239 INFO:tasks.rados.rados.0.trial169.stdout:205: done (4 left) 2026-02-25T08:42:35.239 INFO:tasks.rados.rados.0.trial169.stdout:206: read oid 17 snap 14 2026-02-25T08:42:35.239 INFO:tasks.rados.rados.0.trial169.stdout:206: expect (ObjNum 68 snap 5 seq_num 68) 2026-02-25T08:42:35.239 INFO:tasks.rados.rados.0.trial169.stdout:207: read oid 22 snap -1 2026-02-25T08:42:35.239 INFO:tasks.rados.rados.0.trial169.stdout:207: expect (ObjNum 21 snap 0 seq_num 21) 2026-02-25T08:42:35.239 INFO:tasks.rados.rados.0.trial169.stdout:208: read oid 13 snap -1 2026-02-25T08:42:35.239 INFO:tasks.rados.rados.0.trial169.stdout:208: expect (ObjNum 108 snap 13 seq_num 108) 2026-02-25T08:42:35.240 INFO:tasks.rados.rados.0.trial169.stdout:209: write oid 12 current snap is 17 2026-02-25T08:42:35.240 INFO:tasks.rados.rados.0.trial169.stdout:209: seq_num 123 ranges {467261=717275,1847991=219270} 2026-02-25T08:42:35.243 INFO:tasks.rados.rados.0.trial169.stdout:209: writing trial16941936-12 from 467261 to 1184536 tid 1 2026-02-25T08:42:35.247 INFO:tasks.rados.rados.0.trial169.stdout:209: writing trial16941936-12 from 1847991 to 2067261 tid 2 2026-02-25T08:42:35.247 INFO:tasks.rados.rados.0.trial169.stdout:210: snap_create 2026-02-25T08:42:35.247 INFO:tasks.rados.rados.0.trial169.stdout:201: finishing write tid 2 to trial16941936-24 2026-02-25T08:42:35.247 INFO:tasks.rados.rados.0.trial169.stdout:200: finishing write tid 1 to trial16941936-50 2026-02-25T08:42:35.250 INFO:tasks.rados.rados.0.trial169.stdout:201: finishing write tid 3 to trial16941936-24 2026-02-25T08:42:35.252 INFO:tasks.rados.rados.0.trial169.stdout:201: finishing write tid 4 to trial16941936-24 2026-02-25T08:42:35.252 INFO:tasks.rados.rados.0.trial169.stdout:201: oid 24 updating version 0 to 26 2026-02-25T08:42:35.252 INFO:tasks.rados.rados.0.trial169.stdout:201: oid 24 version 26 is already newer than 26 2026-02-25T08:42:35.252 INFO:tasks.rados.rados.0.trial169.stdout:201: oid 24 version 26 is already newer than 26 2026-02-25T08:42:35.253 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 24 v 26 (ObjNum 119 snap 17 seq_num 119) dirty exists 2026-02-25T08:42:35.253 INFO:tasks.rados.rados.0.trial169.stdout:201: left oid 24 (ObjNum 119 snap 17 seq_num 119) 2026-02-25T08:42:35.256 INFO:tasks.rados.rados.0.trial169.stdout:207: expect (ObjNum 21 snap 0 seq_num 21) 2026-02-25T08:42:35.260 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1027088 2026-02-25T08:42:35.301 INFO:tasks.rados.rados.0.trial169.stdout:200: finishing write tid 2 to trial16941936-50 2026-02-25T08:42:35.301 INFO:tasks.rados.rados.0.trial169.stdout:208: expect (ObjNum 108 snap 13 seq_num 108) 2026-02-25T08:42:35.301 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:35.333 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:35.365 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:35.397 INFO:tasks.rados.rados.0.trial169.stdout:200: finishing write tid 3 to trial16941936-50 2026-02-25T08:42:35.398 INFO:tasks.rados.rados.0.trial169.stdout:206: expect (ObjNum 68 snap 5 seq_num 68) 2026-02-25T08:42:35.398 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:35.433 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:35.503 INFO:tasks.rados.rados.0.trial169.stdout:209: finishing write tid 1 to trial16941936-12 2026-02-25T08:42:35.503 INFO:tasks.rados.rados.0.trial169.stdout:200: finishing write tid 4 to trial16941936-50 2026-02-25T08:42:35.503 INFO:tasks.rados.rados.0.trial169.stdout:209: finishing write tid 2 to trial16941936-12 2026-02-25T08:42:35.504 INFO:tasks.rados.rados.0.trial169.stdout:200: finishing write tid 5 to trial16941936-50 2026-02-25T08:42:35.504 INFO:tasks.rados.rados.0.trial169.stdout:200: oid 50 updating version 0 to 76 2026-02-25T08:42:35.504 INFO:tasks.rados.rados.0.trial169.stdout:200: oid 50 version 76 is already newer than 76 2026-02-25T08:42:35.504 INFO:tasks.rados.rados.0.trial169.stdout:200: oid 50 version 76 is already newer than 76 2026-02-25T08:42:35.504 INFO:tasks.rados.rados.0.trial169.stdout:200: oid 50 version 76 is already newer than 76 2026-02-25T08:42:35.504 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 50 v 76 (ObjNum 118 snap 17 seq_num 118) dirty exists 2026-02-25T08:42:35.504 INFO:tasks.rados.rados.0.trial169.stdout:200: left oid 50 (ObjNum 118 snap 17 seq_num 118) 2026-02-25T08:42:35.504 INFO:tasks.rados.rados.0.trial169.stdout:209: finishing write tid 3 to trial16941936-12 2026-02-25T08:42:35.504 INFO:tasks.rados.rados.0.trial169.stdout:209: finishing write tid 4 to trial16941936-12 2026-02-25T08:42:35.504 INFO:tasks.rados.rados.0.trial169.stdout:209: oid 12 updating version 0 to 30 2026-02-25T08:42:35.504 INFO:tasks.rados.rados.0.trial169.stdout:209: oid 12 version 30 is already newer than 30 2026-02-25T08:42:35.505 INFO:tasks.rados.rados.0.trial169.stdout:209: oid 12 version 30 is already newer than 30 2026-02-25T08:42:35.505 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 12 v 30 (ObjNum 123 snap 17 seq_num 123) dirty exists 2026-02-25T08:42:35.505 INFO:tasks.rados.rados.0.trial169.stdout:209: left oid 12 (ObjNum 123 snap 17 seq_num 123) 2026-02-25T08:42:35.505 INFO:tasks.rados.rados.0.trial169.stdout:202: finishing write tid 1 to trial16941936-45 2026-02-25T08:42:35.505 INFO:tasks.rados.rados.0.trial169.stdout:202: finishing write tid 2 to trial16941936-45 2026-02-25T08:42:35.505 INFO:tasks.rados.rados.0.trial169.stdout:202: finishing write tid 3 to trial16941936-45 2026-02-25T08:42:35.505 INFO:tasks.rados.rados.0.trial169.stdout:202: finishing write tid 4 to trial16941936-45 2026-02-25T08:42:35.505 INFO:tasks.rados.rados.0.trial169.stdout:202: oid 45 updating version 0 to 58 2026-02-25T08:42:35.505 INFO:tasks.rados.rados.0.trial169.stdout:202: oid 45 version 58 is already newer than 58 2026-02-25T08:42:35.505 INFO:tasks.rados.rados.0.trial169.stdout:202: oid 45 version 58 is already newer than 56 2026-02-25T08:42:35.505 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 45 v 58 (ObjNum 120 snap 17 seq_num 120) dirty exists 2026-02-25T08:42:35.505 INFO:tasks.rados.rados.0.trial169.stdout:202: left oid 45 (ObjNum 120 snap 17 seq_num 120) 2026-02-25T08:42:35.505 INFO:tasks.rados.rados.0.trial169.stdout:203: finishing write tid 1 to trial16941936-37 2026-02-25T08:42:35.506 INFO:tasks.rados.rados.0.trial169.stdout:203: finishing write tid 2 to trial16941936-37 2026-02-25T08:42:35.506 INFO:tasks.rados.rados.0.trial169.stdout:203: finishing write tid 3 to trial16941936-37 2026-02-25T08:42:35.506 INFO:tasks.rados.rados.0.trial169.stdout:203: finishing write tid 4 to trial16941936-37 2026-02-25T08:42:35.506 INFO:tasks.rados.rados.0.trial169.stdout:203: oid 37 updating version 0 to 62 2026-02-25T08:42:35.506 INFO:tasks.rados.rados.0.trial169.stdout:203: oid 37 version 62 is already newer than 62 2026-02-25T08:42:35.506 INFO:tasks.rados.rados.0.trial169.stdout:203: oid 37 version 62 is already newer than 60 2026-02-25T08:42:35.506 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 37 v 62 (ObjNum 121 snap 17 seq_num 121) dirty exists 2026-02-25T08:42:35.506 INFO:tasks.rados.rados.0.trial169.stdout:203: left oid 37 (ObjNum 121 snap 17 seq_num 121) 2026-02-25T08:42:36.181 INFO:tasks.rados.rados.0.trial169.stdout:200: done (8 left) 2026-02-25T08:42:36.181 INFO:tasks.rados.rados.0.trial169.stdout:201: done (7 left) 2026-02-25T08:42:36.181 INFO:tasks.rados.rados.0.trial169.stdout:202: done (6 left) 2026-02-25T08:42:36.181 INFO:tasks.rados.rados.0.trial169.stdout:203: done (5 left) 2026-02-25T08:42:36.181 INFO:tasks.rados.rados.0.trial169.stdout:206: done (4 left) 2026-02-25T08:42:36.181 INFO:tasks.rados.rados.0.trial169.stdout:207: done (3 left) 2026-02-25T08:42:36.181 INFO:tasks.rados.rados.0.trial169.stdout:208: done (2 left) 2026-02-25T08:42:36.181 INFO:tasks.rados.rados.0.trial169.stdout:209: done (1 left) 2026-02-25T08:42:36.181 INFO:tasks.rados.rados.0.trial169.stdout:210: done (0 left) 2026-02-25T08:42:36.182 INFO:tasks.rados.rados.0.trial169.stdout:211: read oid 24 snap 17 2026-02-25T08:42:36.182 INFO:tasks.rados.rados.0.trial169.stdout:211: expect (ObjNum 119 snap 17 seq_num 119) 2026-02-25T08:42:36.182 INFO:tasks.rados.rados.0.trial169.stdout:212: write (excl) oid 44 current snap is 18 2026-02-25T08:42:36.182 INFO:tasks.rados.rados.0.trial169.stdout:212: seq_num 124 ranges {484068=792524,1906473=177595} 2026-02-25T08:42:36.185 INFO:tasks.rados.rados.0.trial169.stdout:212: writing trial16941936-44 from 484068 to 1276592 tid 1 2026-02-25T08:42:36.188 INFO:tasks.rados.rados.0.trial169.stdout:212: writing trial16941936-44 from 1906473 to 2084068 tid 2 2026-02-25T08:42:36.188 INFO:tasks.rados.rados.0.trial169.stdout:213: snap_remove snap 14 2026-02-25T08:42:36.967 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:37.046 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:37.043+0000 7fa3acf5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:37.048 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:37.047+0000 7fa3acf5c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:37.052 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:37.091 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:37.181 INFO:tasks.rados.rados.0.trial169.stdout:213: done (2 left) 2026-02-25T08:42:37.181 INFO:tasks.rados.rados.0.trial169.stdout:214: delete oid 10 current snap is 18 2026-02-25T08:42:37.181 INFO:tasks.rados.rados.0.trial169.stdout:211: expect (ObjNum 119 snap 17 seq_num 119) 2026-02-25T08:42:37.205 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:37.256 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1671173 2026-02-25T08:42:37.259 INFO:tasks.rados.rados.0.trial169.stdout:211: done (2 left) 2026-02-25T08:42:37.259 INFO:tasks.rados.rados.0.trial169.stdout:214: done (1 left) 2026-02-25T08:42:37.259 INFO:tasks.rados.rados.0.trial169.stdout:215: snap_create 2026-02-25T08:42:37.260 INFO:tasks.rados.rados.0.trial169.stdout:212: finishing write tid 1 to trial16941936-44 2026-02-25T08:42:37.260 INFO:tasks.rados.rados.0.trial169.stdout:212: finishing write tid 2 to trial16941936-44 2026-02-25T08:42:37.260 INFO:tasks.rados.rados.0.trial169.stdout:212: finishing write tid 3 to trial16941936-44 2026-02-25T08:42:37.260 INFO:tasks.rados.rados.0.trial169.stdout:212: finishing write tid 4 to trial16941936-44 2026-02-25T08:42:37.260 INFO:tasks.rados.rados.0.trial169.stdout:212: oid 44 updating version 0 to 45 2026-02-25T08:42:37.260 INFO:tasks.rados.rados.0.trial169.stdout:212: oid 44 version 45 is already newer than 45 2026-02-25T08:42:37.260 INFO:tasks.rados.rados.0.trial169.stdout:212: oid 44 version 45 is already newer than 43 2026-02-25T08:42:37.260 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 44 v 45 (ObjNum 124 snap 18 seq_num 124) dirty exists 2026-02-25T08:42:37.260 INFO:tasks.rados.rados.0.trial169.stdout:212: left oid 44 (ObjNum 124 snap 18 seq_num 124) 2026-02-25T08:42:37.876 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:42:37.950 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:37.953 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:37.953 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:42:38.187 INFO:tasks.rados.rados.0.trial169.stdout:212: done (1 left) 2026-02-25T08:42:38.187 INFO:tasks.rados.rados.0.trial169.stdout:215: done (0 left) 2026-02-25T08:42:38.187 INFO:tasks.rados.rados.0.trial169.stdout:216: read oid 46 snap -1 2026-02-25T08:42:38.188 INFO:tasks.rados.rados.0.trial169.stdout:216: expect (ObjNum 79 snap 8 seq_num 79) 2026-02-25T08:42:38.188 INFO:tasks.rados.rados.0.trial169.stdout:217: snap_create 2026-02-25T08:42:38.220 INFO:tasks.rados.rados.0.trial169.stdout:216: expect (ObjNum 79 snap 8 seq_num 79) 2026-02-25T08:42:38.220 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:38.297 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:39.045 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~33s 2026-02-25T08:42:39.092 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:39.172 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:39.169+0000 7f941303d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:39.174 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:39.173+0000 7f941303d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:39.178 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:39.187 INFO:tasks.rados.rados.0.trial169.stdout:216: done (1 left) 2026-02-25T08:42:39.187 INFO:tasks.rados.rados.0.trial169.stdout:217: done (0 left) 2026-02-25T08:42:39.187 INFO:tasks.rados.rados.0.trial169.stdout:218: delete oid 4 current snap is 20 2026-02-25T08:42:39.233 INFO:tasks.rados.rados.0.trial169.stdout:218: done (0 left) 2026-02-25T08:42:39.233 INFO:tasks.rados.rados.0.trial169.stdout:219: snap_remove snap 17 2026-02-25T08:42:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:39.242 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:40.193 INFO:tasks.rados.rados.0.trial169.stdout:219: done (0 left) 2026-02-25T08:42:40.193 INFO:tasks.rados.rados.0.trial169.stdout:220: snap_create 2026-02-25T08:42:41.195 INFO:tasks.rados.rados.0.trial169.stdout:220: done (0 left) 2026-02-25T08:42:41.195 INFO:tasks.rados.rados.0.trial169.stdout:221: write oid 32 current snap is 21 2026-02-25T08:42:41.196 INFO:tasks.rados.rados.0.trial169.stdout:221: seq_num 127 ranges {534489=618271,1565566=568923} 2026-02-25T08:42:41.199 INFO:tasks.rados.rados.0.trial169.stdout:221: writing trial16941936-32 from 534489 to 1152760 tid 1 2026-02-25T08:42:41.203 INFO:tasks.rados.rados.0.trial169.stdout:221: writing trial16941936-32 from 1565566 to 2134489 tid 2 2026-02-25T08:42:41.204 INFO:tasks.rados.rados.0.trial169.stdout:222: read oid 36 snap 18 2026-02-25T08:42:41.204 INFO:tasks.rados.rados.0.trial169.stdout:222: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:42:41.204 INFO:tasks.rados.rados.0.trial169.stdout:223: read oid 4 snap -1 2026-02-25T08:42:41.204 INFO:tasks.rados.rados.0.trial169.stdout:223: expect deleted 2026-02-25T08:42:41.204 INFO:tasks.rados.rados.0.trial169.stdout:224: snap_create 2026-02-25T08:42:41.224 INFO:tasks.rados.rados.0.trial169.stdout:222: expect (ObjNum 35 snap 0 seq_num 35) 2026-02-25T08:42:41.243 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:41.286 INFO:tasks.rados.rados.0.trial169.stdout:221: finishing write tid 1 to trial16941936-32 2026-02-25T08:42:41.286 INFO:tasks.rados.rados.0.trial169.stdout:221: finishing write tid 2 to trial16941936-32 2026-02-25T08:42:41.286 INFO:tasks.rados.rados.0.trial169.stdout:221: finishing write tid 3 to trial16941936-32 2026-02-25T08:42:41.287 INFO:tasks.rados.rados.0.trial169.stdout:221: finishing write tid 4 to trial16941936-32 2026-02-25T08:42:41.287 INFO:tasks.rados.rados.0.trial169.stdout:221: oid 32 updating version 0 to 30 2026-02-25T08:42:41.287 INFO:tasks.rados.rados.0.trial169.stdout:221: oid 32 version 30 is already newer than 30 2026-02-25T08:42:41.287 INFO:tasks.rados.rados.0.trial169.stdout:221: oid 32 version 30 is already newer than 30 2026-02-25T08:42:41.287 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 32 v 30 (ObjNum 127 snap 21 seq_num 127) dirty exists 2026-02-25T08:42:41.287 INFO:tasks.rados.rados.0.trial169.stdout:221: left oid 32 (ObjNum 127 snap 21 seq_num 127) 2026-02-25T08:42:41.321 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:41.318+0000 7fa73cd8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:41.323 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:41.321+0000 7fa73cd8c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:41.327 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:41.367 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:41.367 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:42.200 INFO:tasks.rados.rados.0.trial169.stdout:221: done (3 left) 2026-02-25T08:42:42.201 INFO:tasks.rados.rados.0.trial169.stdout:222: done (2 left) 2026-02-25T08:42:42.201 INFO:tasks.rados.rados.0.trial169.stdout:223: done (1 left) 2026-02-25T08:42:42.201 INFO:tasks.rados.rados.0.trial169.stdout:224: done (0 left) 2026-02-25T08:42:42.201 INFO:tasks.rados.rados.0.trial169.stdout:225: read oid 29 snap -1 2026-02-25T08:42:42.201 INFO:tasks.rados.rados.0.trial169.stdout:225: expect deleted 2026-02-25T08:42:42.201 INFO:tasks.rados.rados.0.trial169.stdout:226: snap_create 2026-02-25T08:42:42.954 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:42:43.026 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:43.030 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:42:43.202 INFO:tasks.rados.rados.0.trial169.stdout:225: done (1 left) 2026-02-25T08:42:43.203 INFO:tasks.rados.rados.0.trial169.stdout:226: done (0 left) 2026-02-25T08:42:43.203 INFO:tasks.rados.rados.0.trial169.stdout:227: write (excl) oid 4 current snap is 23 2026-02-25T08:42:43.203 INFO:tasks.rados.rados.0.trial169.stdout:227: seq_num 128 ranges {551296=693520,2024048=127248} 2026-02-25T08:42:43.207 INFO:tasks.rados.rados.0.trial169.stdout:227: writing trial16941936-4 from 551296 to 1244816 tid 1 2026-02-25T08:42:43.211 INFO:tasks.rados.rados.0.trial169.stdout:227: writing trial16941936-4 from 2024048 to 2151296 tid 2 2026-02-25T08:42:43.211 INFO:tasks.rados.rados.0.trial169.stdout:228: write oid 37 current snap is 23 2026-02-25T08:42:43.211 INFO:tasks.rados.rados.0.trial169.stdout:228: seq_num 129 ranges {568103=768769,2082530=85573} 2026-02-25T08:42:43.215 INFO:tasks.rados.rados.0.trial169.stdout:228: writing trial16941936-37 from 568103 to 1336872 tid 1 2026-02-25T08:42:43.219 INFO:tasks.rados.rados.0.trial169.stdout:228: writing trial16941936-37 from 2082530 to 2168103 tid 2 2026-02-25T08:42:43.219 INFO:tasks.rados.rados.0.trial169.stdout:229: delete oid 35 current snap is 23 2026-02-25T08:42:43.239 INFO:tasks.rados.rados.0.trial169.stdout:227: finishing write tid 1 to trial16941936-4 2026-02-25T08:42:43.255 INFO:tasks.rados.rados.0.trial169.stdout:227: finishing write tid 2 to trial16941936-4 2026-02-25T08:42:43.262 INFO:tasks.rados.rados.0.trial169.stdout:227: finishing write tid 3 to trial16941936-4 2026-02-25T08:42:43.265 INFO:tasks.rados.rados.0.trial169.stdout:227: finishing write tid 4 to trial16941936-4 2026-02-25T08:42:43.265 INFO:tasks.rados.rados.0.trial169.stdout:227: oid 4 updating version 0 to 87 2026-02-25T08:42:43.265 INFO:tasks.rados.rados.0.trial169.stdout:227: oid 4 version 87 is already newer than 85 2026-02-25T08:42:43.265 INFO:tasks.rados.rados.0.trial169.stdout:227: oid 4 version 87 is already newer than 87 2026-02-25T08:42:43.265 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 4 v 87 (ObjNum 128 snap 23 seq_num 128) dirty exists 2026-02-25T08:42:43.265 INFO:tasks.rados.rados.0.trial169.stdout:227: left oid 4 (ObjNum 128 snap 23 seq_num 128) 2026-02-25T08:42:43.286 INFO:tasks.rados.rados.0.trial169.stdout:228: finishing write tid 1 to trial16941936-37 2026-02-25T08:42:43.293 INFO:tasks.rados.rados.0.trial169.stdout:228: finishing write tid 2 to trial16941936-37 2026-02-25T08:42:43.297 INFO:tasks.rados.rados.0.trial169.stdout:228: finishing write tid 3 to trial16941936-37 2026-02-25T08:42:43.299 INFO:tasks.rados.rados.0.trial169.stdout:228: finishing write tid 4 to trial16941936-37 2026-02-25T08:42:43.299 INFO:tasks.rados.rados.0.trial169.stdout:228: oid 37 updating version 0 to 66 2026-02-25T08:42:43.299 INFO:tasks.rados.rados.0.trial169.stdout:228: oid 37 version 66 is already newer than 66 2026-02-25T08:42:43.299 INFO:tasks.rados.rados.0.trial169.stdout:228: oid 37 version 66 is already newer than 66 2026-02-25T08:42:43.299 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 37 v 66 (ObjNum 129 snap 23 seq_num 129) dirty exists 2026-02-25T08:42:43.299 INFO:tasks.rados.rados.0.trial169.stdout:228: left oid 37 (ObjNum 129 snap 23 seq_num 129) 2026-02-25T08:42:43.309 INFO:tasks.rados.rados.0.trial169.stdout:227: done (2 left) 2026-02-25T08:42:43.309 INFO:tasks.rados.rados.0.trial169.stdout:228: done (1 left) 2026-02-25T08:42:43.309 INFO:tasks.rados.rados.0.trial169.stdout:229: done (0 left) 2026-02-25T08:42:43.309 INFO:tasks.rados.rados.0.trial169.stdout:230: delete oid 35 current snap is 23 2026-02-25T08:42:43.311 INFO:tasks.rados.rados.0.trial169.stdout:230: done (0 left) 2026-02-25T08:42:43.311 INFO:tasks.rados.rados.0.trial169.stdout:231: delete oid 46 current snap is 23 2026-02-25T08:42:43.334 INFO:tasks.rados.rados.0.trial169.stdout:231: done (0 left) 2026-02-25T08:42:43.334 INFO:tasks.rados.rados.0.trial169.stdout:232: write oid 26 current snap is 23 2026-02-25T08:42:43.334 INFO:tasks.rados.rados.0.trial169.stdout:232: seq_num 133 ranges {635331=786118,1916458=318873} 2026-02-25T08:42:43.340 INFO:tasks.rados.rados.0.trial169.stdout:232: writing trial16941936-26 from 635331 to 1421449 tid 1 2026-02-25T08:42:43.343 INFO:tasks.rados.rados.0.trial169.stdout:232: writing trial16941936-26 from 1916458 to 2235331 tid 2 2026-02-25T08:42:43.343 INFO:tasks.rados.rados.0.trial169.stdout:233: snap_remove snap 20 2026-02-25T08:42:43.368 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:43.446 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:43.443+0000 7f0d01f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:43.449 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:43.447+0000 7f0d01f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:43.452 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:43.491 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:43.491 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:44.207 INFO:tasks.rados.rados.0.trial169.stdout:233: done (1 left) 2026-02-25T08:42:44.207 INFO:tasks.rados.rados.0.trial169.stdout:234: snap_create 2026-02-25T08:42:44.208 INFO:tasks.rados.rados.0.trial169.stdout:232: finishing write tid 1 to trial16941936-26 2026-02-25T08:42:44.208 INFO:tasks.rados.rados.0.trial169.stdout:232: finishing write tid 2 to trial16941936-26 2026-02-25T08:42:44.208 INFO:tasks.rados.rados.0.trial169.stdout:232: finishing write tid 3 to trial16941936-26 2026-02-25T08:42:44.208 INFO:tasks.rados.rados.0.trial169.stdout:232: finishing write tid 4 to trial16941936-26 2026-02-25T08:42:44.208 INFO:tasks.rados.rados.0.trial169.stdout:232: oid 26 updating version 0 to 65 2026-02-25T08:42:44.208 INFO:tasks.rados.rados.0.trial169.stdout:232: oid 26 version 65 is already newer than 65 2026-02-25T08:42:44.208 INFO:tasks.rados.rados.0.trial169.stdout:232: oid 26 version 65 is already newer than 65 2026-02-25T08:42:44.208 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 26 v 65 (ObjNum 133 snap 23 seq_num 133) dirty exists 2026-02-25T08:42:44.208 INFO:tasks.rados.rados.0.trial169.stdout:232: left oid 26 (ObjNum 133 snap 23 seq_num 133) 2026-02-25T08:42:44.548 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~39s 2026-02-25T08:42:45.209 INFO:tasks.rados.rados.0.trial169.stdout:232: done (1 left) 2026-02-25T08:42:45.209 INFO:tasks.rados.rados.0.trial169.stdout:234: done (0 left) 2026-02-25T08:42:45.209 INFO:tasks.rados.rados.0.trial169.stdout:235: write (excl) oid 29 current snap is 24 2026-02-25T08:42:45.209 INFO:tasks.rados.rados.0.trial169.stdout:235: seq_num 134 ranges {652138=461367,1574940=677198} 2026-02-25T08:42:45.213 INFO:tasks.rados.rados.0.trial169.stdout:235: writing trial16941936-29 from 652138 to 1113505 tid 1 2026-02-25T08:42:45.218 INFO:tasks.rados.rados.0.trial169.stdout:235: writing trial16941936-29 from 1574940 to 2252138 tid 2 2026-02-25T08:42:45.218 INFO:tasks.rados.rados.0.trial169.stdout:236: snap_remove snap 18 2026-02-25T08:42:45.492 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:45.569 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:45.566+0000 7fb1c7964640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:45.571 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:45.570+0000 7fb1c7964640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:45.574 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:45.614 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:45.614 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:46.215 INFO:tasks.rados.rados.0.trial169.stdout:236: done (1 left) 2026-02-25T08:42:46.215 INFO:tasks.rados.rados.0.trial169.stdout:237: write (excl) oid 28 current snap is 24 2026-02-25T08:42:46.215 INFO:tasks.rados.rados.0.trial169.stdout:235: finishing write tid 1 to trial16941936-29 2026-02-25T08:42:46.215 INFO:tasks.rados.rados.0.trial169.stdout:237: seq_num 135 ranges {668945=536616,1633422=491363} 2026-02-25T08:42:46.219 INFO:tasks.rados.rados.0.trial169.stdout:237: writing trial16941936-28 from 668945 to 1205561 tid 1 2026-02-25T08:42:46.223 INFO:tasks.rados.rados.0.trial169.stdout:237: writing trial16941936-28 from 1633422 to 2124785 tid 2 2026-02-25T08:42:46.223 INFO:tasks.rados.rados.0.trial169.stdout:238: write oid 19 current snap is 24 2026-02-25T08:42:46.223 INFO:tasks.rados.rados.0.trial169.stdout:235: finishing write tid 2 to trial16941936-29 2026-02-25T08:42:46.223 INFO:tasks.rados.rados.0.trial169.stdout:235: finishing write tid 3 to trial16941936-29 2026-02-25T08:42:46.223 INFO:tasks.rados.rados.0.trial169.stdout:235: finishing write tid 4 to trial16941936-29 2026-02-25T08:42:46.223 INFO:tasks.rados.rados.0.trial169.stdout:235: oid 29 updating version 0 to 94 2026-02-25T08:42:46.223 INFO:tasks.rados.rados.0.trial169.stdout:235: oid 29 version 94 is already newer than 94 2026-02-25T08:42:46.224 INFO:tasks.rados.rados.0.trial169.stdout:235: oid 29 version 94 is already newer than 92 2026-02-25T08:42:46.224 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 29 v 94 (ObjNum 134 snap 24 seq_num 134) dirty exists 2026-02-25T08:42:46.224 INFO:tasks.rados.rados.0.trial169.stdout:235: left oid 29 (ObjNum 134 snap 24 seq_num 134) 2026-02-25T08:42:46.224 INFO:tasks.rados.rados.0.trial169.stdout:238: seq_num 136 ranges {685752=611865,1975551=310201} 2026-02-25T08:42:46.227 INFO:tasks.rados.rados.0.trial169.stdout:238: writing trial16941936-19 from 685752 to 1297617 tid 1 2026-02-25T08:42:46.231 INFO:tasks.rados.rados.0.trial169.stdout:238: writing trial16941936-19 from 1975551 to 2285752 tid 2 2026-02-25T08:42:46.231 INFO:tasks.rados.rados.0.trial169.stdout:235: done (2 left) 2026-02-25T08:42:46.232 INFO:tasks.rados.rados.0.trial169.stdout:239: read oid 14 snap 22 2026-02-25T08:42:46.232 INFO:tasks.rados.rados.0.trial169.stdout:239: expect deleted 2026-02-25T08:42:46.232 INFO:tasks.rados.rados.0.trial169.stdout:240: read oid 29 snap -1 2026-02-25T08:42:46.232 INFO:tasks.rados.rados.0.trial169.stdout:240: expect (ObjNum 134 snap 24 seq_num 134) 2026-02-25T08:42:46.232 INFO:tasks.rados.rados.0.trial169.stdout:241: read oid 46 snap -1 2026-02-25T08:42:46.232 INFO:tasks.rados.rados.0.trial169.stdout:241: expect deleted 2026-02-25T08:42:46.232 INFO:tasks.rados.rados.0.trial169.stdout:242: snap_create 2026-02-25T08:42:46.249 INFO:tasks.rados.rados.0.trial169.stdout:237: finishing write tid 1 to trial16941936-28 2026-02-25T08:42:46.265 INFO:tasks.rados.rados.0.trial169.stdout:237: finishing write tid 2 to trial16941936-28 2026-02-25T08:42:46.270 INFO:tasks.rados.rados.0.trial169.stdout:237: finishing write tid 3 to trial16941936-28 2026-02-25T08:42:46.272 INFO:tasks.rados.rados.0.trial169.stdout:237: finishing write tid 4 to trial16941936-28 2026-02-25T08:42:46.272 INFO:tasks.rados.rados.0.trial169.stdout:237: oid 28 updating version 0 to 27 2026-02-25T08:42:46.272 INFO:tasks.rados.rados.0.trial169.stdout:237: oid 28 updating version 27 to 29 2026-02-25T08:42:46.272 INFO:tasks.rados.rados.0.trial169.stdout:237: oid 28 version 29 is already newer than 29 2026-02-25T08:42:46.272 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 28 v 29 (ObjNum 135 snap 24 seq_num 135) dirty exists 2026-02-25T08:42:46.272 INFO:tasks.rados.rados.0.trial169.stdout:237: left oid 28 (ObjNum 135 snap 24 seq_num 135) 2026-02-25T08:42:46.279 INFO:tasks.rados.rados.0.trial169.stdout:240: expect (ObjNum 134 snap 24 seq_num 134) 2026-02-25T08:42:46.280 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 259672 2026-02-25T08:42:46.316 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:46.335 INFO:tasks.rados.rados.0.trial169.stdout:238: finishing write tid 1 to trial16941936-19 2026-02-25T08:42:46.335 INFO:tasks.rados.rados.0.trial169.stdout:238: finishing write tid 2 to trial16941936-19 2026-02-25T08:42:46.335 INFO:tasks.rados.rados.0.trial169.stdout:238: finishing write tid 3 to trial16941936-19 2026-02-25T08:42:46.335 INFO:tasks.rados.rados.0.trial169.stdout:238: finishing write tid 4 to trial16941936-19 2026-02-25T08:42:46.335 INFO:tasks.rados.rados.0.trial169.stdout:238: oid 19 updating version 0 to 38 2026-02-25T08:42:46.335 INFO:tasks.rados.rados.0.trial169.stdout:238: oid 19 version 38 is already newer than 38 2026-02-25T08:42:46.335 INFO:tasks.rados.rados.0.trial169.stdout:238: oid 19 version 38 is already newer than 38 2026-02-25T08:42:46.335 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 19 v 38 (ObjNum 136 snap 24 seq_num 136) dirty exists 2026-02-25T08:42:46.336 INFO:tasks.rados.rados.0.trial169.stdout:238: left oid 19 (ObjNum 136 snap 24 seq_num 136) 2026-02-25T08:42:47.215 INFO:tasks.rados.rados.0.trial169.stdout:237: done (5 left) 2026-02-25T08:42:47.216 INFO:tasks.rados.rados.0.trial169.stdout:238: done (4 left) 2026-02-25T08:42:47.216 INFO:tasks.rados.rados.0.trial169.stdout:239: done (3 left) 2026-02-25T08:42:47.216 INFO:tasks.rados.rados.0.trial169.stdout:240: done (2 left) 2026-02-25T08:42:47.216 INFO:tasks.rados.rados.0.trial169.stdout:241: done (1 left) 2026-02-25T08:42:47.216 INFO:tasks.rados.rados.0.trial169.stdout:242: done (0 left) 2026-02-25T08:42:47.216 INFO:tasks.rados.rados.0.trial169.stdout:243: snap_create 2026-02-25T08:42:47.615 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:47.694 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:47.692+0000 7f358df6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:47.697 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:47.695+0000 7f358df6f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:47.700 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:47.740 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:47.740 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:48.031 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:42:48.102 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:48.105 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:42:48.119 INFO:tasks.rados.rados.0.trial169.stdout:243: done (0 left) 2026-02-25T08:42:48.119 INFO:tasks.rados.rados.0.trial169.stdout:244: write oid 41 current snap is 26 2026-02-25T08:42:48.119 INFO:tasks.rados.rados.0.trial169.stdout:244: seq_num 137 ranges {702559=403467,1750386=495032} 2026-02-25T08:42:48.123 INFO:tasks.rados.rados.0.trial169.stdout:244: writing trial16941936-41 from 702559 to 1106026 tid 1 2026-02-25T08:42:48.127 INFO:tasks.rados.rados.0.trial169.stdout:244: writing trial16941936-41 from 1750386 to 2245418 tid 2 2026-02-25T08:42:48.128 INFO:tasks.rados.rados.0.trial169.stdout:245: snap_remove snap 22 2026-02-25T08:42:49.120 INFO:tasks.rados.rados.0.trial169.stdout:245: done (1 left) 2026-02-25T08:42:49.120 INFO:tasks.rados.rados.0.trial169.stdout:246: snap_create 2026-02-25T08:42:49.120 INFO:tasks.rados.rados.0.trial169.stdout:244: finishing write tid 1 to trial16941936-41 2026-02-25T08:42:49.120 INFO:tasks.rados.rados.0.trial169.stdout:244: finishing write tid 2 to trial16941936-41 2026-02-25T08:42:49.120 INFO:tasks.rados.rados.0.trial169.stdout:244: finishing write tid 3 to trial16941936-41 2026-02-25T08:42:49.120 INFO:tasks.rados.rados.0.trial169.stdout:244: finishing write tid 4 to trial16941936-41 2026-02-25T08:42:49.120 INFO:tasks.rados.rados.0.trial169.stdout:244: oid 41 updating version 0 to 73 2026-02-25T08:42:49.120 INFO:tasks.rados.rados.0.trial169.stdout:244: oid 41 version 73 is already newer than 73 2026-02-25T08:42:49.121 INFO:tasks.rados.rados.0.trial169.stdout:244: oid 41 version 73 is already newer than 73 2026-02-25T08:42:49.121 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 41 v 73 (ObjNum 137 snap 26 seq_num 137) dirty exists 2026-02-25T08:42:49.121 INFO:tasks.rados.rados.0.trial169.stdout:244: left oid 41 (ObjNum 137 snap 26 seq_num 137) 2026-02-25T08:42:49.741 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:49.819 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:49.816+0000 7f712e0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:49.821 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:49.820+0000 7f712e0cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:49.825 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:49.863 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:49.863 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:50.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~44s 2026-02-25T08:42:50.122 INFO:tasks.rados.rados.0.trial169.stdout:244: done (1 left) 2026-02-25T08:42:50.122 INFO:tasks.rados.rados.0.trial169.stdout:246: done (0 left) 2026-02-25T08:42:50.123 INFO:tasks.rados.rados.0.trial169.stdout:247: delete oid 33 current snap is 27 2026-02-25T08:42:50.158 INFO:tasks.rados.rados.0.trial169.stdout:247: done (0 left) 2026-02-25T08:42:50.158 INFO:tasks.rados.rados.0.trial169.stdout:248: snap_create 2026-02-25T08:42:51.128 INFO:tasks.rados.rados.0.trial169.stdout:248: done (0 left) 2026-02-25T08:42:51.128 INFO:tasks.rados.rados.0.trial169.stdout:249: write (excl) oid 13 current snap is 28 2026-02-25T08:42:51.128 INFO:tasks.rados.rados.0.trial169.stdout:249: seq_num 139 ranges {736173=553965,1867350=468823} 2026-02-25T08:42:51.132 INFO:tasks.rados.rados.0.trial169.stdout:249: writing trial16941936-13 from 736173 to 1290138 tid 1 2026-02-25T08:42:51.136 INFO:tasks.rados.rados.0.trial169.stdout:249: writing trial16941936-13 from 1867350 to 2336173 tid 2 2026-02-25T08:42:51.136 INFO:tasks.rados.rados.0.trial169.stdout:250: snap_create 2026-02-25T08:42:51.171 INFO:tasks.rados.rados.0.trial169.stdout:249: finishing write tid 1 to trial16941936-13 2026-02-25T08:42:51.184 INFO:tasks.rados.rados.0.trial169.stdout:249: finishing write tid 2 to trial16941936-13 2026-02-25T08:42:51.189 INFO:tasks.rados.rados.0.trial169.stdout:249: finishing write tid 3 to trial16941936-13 2026-02-25T08:42:51.191 INFO:tasks.rados.rados.0.trial169.stdout:249: finishing write tid 4 to trial16941936-13 2026-02-25T08:42:51.191 INFO:tasks.rados.rados.0.trial169.stdout:249: oid 13 updating version 0 to 81 2026-02-25T08:42:51.191 INFO:tasks.rados.rados.0.trial169.stdout:249: oid 13 version 81 is already newer than 79 2026-02-25T08:42:51.191 INFO:tasks.rados.rados.0.trial169.stdout:249: oid 13 version 81 is already newer than 81 2026-02-25T08:42:51.191 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 13 v 81 (ObjNum 139 snap 28 seq_num 139) dirty exists 2026-02-25T08:42:51.191 INFO:tasks.rados.rados.0.trial169.stdout:249: left oid 13 (ObjNum 139 snap 28 seq_num 139) 2026-02-25T08:42:51.864 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:51.942 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:51.939+0000 7f08e8ba4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:51.944 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:51.943+0000 7f08e8ba4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:51.948 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:51.987 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:51.988 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:52.132 INFO:tasks.rados.rados.0.trial169.stdout:249: done (1 left) 2026-02-25T08:42:52.133 INFO:tasks.rados.rados.0.trial169.stdout:250: done (0 left) 2026-02-25T08:42:52.133 INFO:tasks.rados.rados.0.trial169.stdout:251: read oid 49 snap -1 2026-02-25T08:42:52.133 INFO:tasks.rados.rados.0.trial169.stdout:251: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:42:52.133 INFO:tasks.rados.rados.0.trial169.stdout:252: read oid 9 snap -1 2026-02-25T08:42:52.133 INFO:tasks.rados.rados.0.trial169.stdout:252: expect (ObjNum 113 snap 16 seq_num 113) 2026-02-25T08:42:52.133 INFO:tasks.rados.rados.0.trial169.stdout:253: snap_remove snap 23 2026-02-25T08:42:53.107 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:42:53.133 INFO:tasks.rados.rados.0.trial169.stdout:253: done (2 left) 2026-02-25T08:42:53.133 INFO:tasks.rados.rados.0.trial169.stdout:254: delete oid 33 current snap is 29 2026-02-25T08:42:53.134 INFO:tasks.rados.rados.0.trial169.stdout:251: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:42:53.134 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:53.157 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:53.181 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:53.181 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 125397 2026-02-25T08:42:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:53.184 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:42:53.204 INFO:tasks.rados.rados.0.trial169.stdout:252: expect (ObjNum 113 snap 16 seq_num 113) 2026-02-25T08:42:53.204 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:53.336 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 850586 2026-02-25T08:42:53.392 INFO:tasks.rados.rados.0.trial169.stdout:251: done (2 left) 2026-02-25T08:42:53.392 INFO:tasks.rados.rados.0.trial169.stdout:252: done (1 left) 2026-02-25T08:42:53.392 INFO:tasks.rados.rados.0.trial169.stdout:254: done (0 left) 2026-02-25T08:42:53.392 INFO:tasks.rados.rados.0.trial169.stdout:255: write (excl) oid 24 current snap is 29 2026-02-25T08:42:53.392 INFO:tasks.rados.rados.0.trial169.stdout:255: seq_num 141 ranges {769787=704463,2267961=101826} 2026-02-25T08:42:53.397 INFO:tasks.rados.rados.0.trial169.stdout:255: writing trial16941936-24 from 769787 to 1474250 tid 1 2026-02-25T08:42:53.401 INFO:tasks.rados.rados.0.trial169.stdout:255: writing trial16941936-24 from 2267961 to 2369787 tid 2 2026-02-25T08:42:53.401 INFO:tasks.rados.rados.0.trial169.stdout:256: read oid 7 snap -1 2026-02-25T08:42:53.401 INFO:tasks.rados.rados.0.trial169.stdout:256: expect (ObjNum 105 snap 13 seq_num 105) 2026-02-25T08:42:53.401 INFO:tasks.rados.rados.0.trial169.stdout:257: read oid 45 snap -1 2026-02-25T08:42:53.401 INFO:tasks.rados.rados.0.trial169.stdout:257: expect (ObjNum 120 snap 17 seq_num 120) 2026-02-25T08:42:53.401 INFO:tasks.rados.rados.0.trial169.stdout:258: snap_remove snap 21 2026-02-25T08:42:53.989 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:54.066 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:54.063+0000 7f2c46c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:54.068 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:54.067+0000 7f2c46c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:54.072 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:54.113 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:54.113 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:54.136 INFO:tasks.rados.rados.0.trial169.stdout:258: done (3 left) 2026-02-25T08:42:54.136 INFO:tasks.rados.rados.0.trial169.stdout:259: delete oid 36 current snap is 29 2026-02-25T08:42:54.136 INFO:tasks.rados.rados.0.trial169.stdout:256: expect (ObjNum 105 snap 13 seq_num 105) 2026-02-25T08:42:54.136 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:54.177 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:42:54.221 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 599436 2026-02-25T08:42:54.254 INFO:tasks.rados.rados.0.trial169.stdout:257: expect (ObjNum 120 snap 17 seq_num 120) 2026-02-25T08:42:54.333 INFO:tasks.rados.rados.0.trial169.stdout:255: finishing write tid 1 to trial16941936-24 2026-02-25T08:42:54.333 INFO:tasks.rados.rados.0.trial169.stdout:256: done (3 left) 2026-02-25T08:42:54.333 INFO:tasks.rados.rados.0.trial169.stdout:257: done (2 left) 2026-02-25T08:42:54.333 INFO:tasks.rados.rados.0.trial169.stdout:259: done (1 left) 2026-02-25T08:42:54.333 INFO:tasks.rados.rados.0.trial169.stdout:260: snap_create 2026-02-25T08:42:54.333 INFO:tasks.rados.rados.0.trial169.stdout:255: finishing write tid 2 to trial16941936-24 2026-02-25T08:42:54.333 INFO:tasks.rados.rados.0.trial169.stdout:255: finishing write tid 3 to trial16941936-24 2026-02-25T08:42:54.333 INFO:tasks.rados.rados.0.trial169.stdout:255: finishing write tid 4 to trial16941936-24 2026-02-25T08:42:54.333 INFO:tasks.rados.rados.0.trial169.stdout:255: oid 24 updating version 0 to 32 2026-02-25T08:42:54.334 INFO:tasks.rados.rados.0.trial169.stdout:255: oid 24 updating version 32 to 34 2026-02-25T08:42:54.334 INFO:tasks.rados.rados.0.trial169.stdout:255: oid 24 version 34 is already newer than 34 2026-02-25T08:42:54.334 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 24 v 34 (ObjNum 141 snap 29 seq_num 141) dirty exists 2026-02-25T08:42:54.334 INFO:tasks.rados.rados.0.trial169.stdout:255: left oid 24 (ObjNum 141 snap 29 seq_num 141) 2026-02-25T08:42:55.141 INFO:tasks.rados.rados.0.trial169.stdout:255: done (1 left) 2026-02-25T08:42:55.141 INFO:tasks.rados.rados.0.trial169.stdout:260: done (0 left) 2026-02-25T08:42:55.142 INFO:tasks.rados.rados.0.trial169.stdout:261: snap_create 2026-02-25T08:42:55.555 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~50s 2026-02-25T08:42:56.114 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:42:56.142 INFO:tasks.rados.rados.0.trial169.stdout:261: done (0 left) 2026-02-25T08:42:56.142 INFO:tasks.rados.rados.0.trial169.stdout:262: snap_remove snap 28 2026-02-25T08:42:56.196 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:56.194+0000 7ff99510b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:56.199 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:56.198+0000 7ff99510b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:56.202 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:56.242 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:56.242 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:57.146 INFO:tasks.rados.rados.0.trial169.stdout:262: done (0 left) 2026-02-25T08:42:57.147 INFO:tasks.rados.rados.0.trial169.stdout:263: snap_create 2026-02-25T08:42:58.122 INFO:tasks.rados.rados.0.trial169.stdout:263: done (0 left) 2026-02-25T08:42:58.122 INFO:tasks.rados.rados.0.trial169.stdout:264: write (excl) oid 4 current snap is 32 2026-02-25T08:42:58.122 INFO:tasks.rados.rados.0.trial169.stdout:264: seq_num 143 ranges {403401=454961,1584925=506039} 2026-02-25T08:42:58.125 INFO:tasks.rados.rados.0.trial169.stdout:264: writing trial16941936-4 from 403401 to 858362 tid 1 2026-02-25T08:42:58.130 INFO:tasks.rados.rados.0.trial169.stdout:264: writing trial16941936-4 from 1584925 to 2090964 tid 2 2026-02-25T08:42:58.130 INFO:tasks.rados.rados.0.trial169.stdout:265: write (excl) oid 22 current snap is 32 2026-02-25T08:42:58.130 INFO:tasks.rados.rados.0.trial169.stdout:265: seq_num 144 ranges {420208=530210,1527054=766050} 2026-02-25T08:42:58.134 INFO:tasks.rados.rados.0.trial169.stdout:265: writing trial16941936-22 from 420208 to 950418 tid 1 2026-02-25T08:42:58.139 INFO:tasks.rados.rados.0.trial169.stdout:265: writing trial16941936-22 from 1527054 to 2293104 tid 2 2026-02-25T08:42:58.140 INFO:tasks.rados.rados.0.trial169.stdout:266: write oid 34 current snap is 32 2026-02-25T08:42:58.140 INFO:tasks.rados.rados.0.trial169.stdout:266: seq_num 145 ranges {437015=721812,1701889=509708} 2026-02-25T08:42:58.143 INFO:tasks.rados.rados.0.trial169.stdout:266: writing trial16941936-34 from 437015 to 1158827 tid 1 2026-02-25T08:42:58.147 INFO:tasks.rados.rados.0.trial169.stdout:266: writing trial16941936-34 from 1701889 to 2211597 tid 2 2026-02-25T08:42:58.148 INFO:tasks.rados.rados.0.trial169.stdout:267: write oid 15 current snap is 32 2026-02-25T08:42:58.148 INFO:tasks.rados.rados.0.trial169.stdout:267: seq_num 146 ranges {453822=797061,1760371=653366} 2026-02-25T08:42:58.152 INFO:tasks.rados.rados.0.trial169.stdout:267: writing trial16941936-15 from 453822 to 1250883 tid 1 2026-02-25T08:42:58.157 INFO:tasks.rados.rados.0.trial169.stdout:267: writing trial16941936-15 from 1760371 to 2413737 tid 2 2026-02-25T08:42:58.157 INFO:tasks.rados.rados.0.trial169.stdout:268: read oid 10 snap 25 2026-02-25T08:42:58.157 INFO:tasks.rados.rados.0.trial169.stdout:268: expect deleted 2026-02-25T08:42:58.157 INFO:tasks.rados.rados.0.trial169.stdout:269: write oid 30 current snap is 32 2026-02-25T08:42:58.157 INFO:tasks.rados.rados.0.trial169.stdout:269: seq_num 147 ranges {470629=472310,1418853=513377} 2026-02-25T08:42:58.160 INFO:tasks.rados.rados.0.trial169.stdout:269: writing trial16941936-30 from 470629 to 942939 tid 1 2026-02-25T08:42:58.164 INFO:tasks.rados.rados.0.trial169.stdout:269: writing trial16941936-30 from 1418853 to 1932230 tid 2 2026-02-25T08:42:58.164 INFO:tasks.rados.rados.0.trial169.stdout:270: delete oid 44 current snap is 32 2026-02-25T08:42:58.179 INFO:tasks.rados.rados.0.trial169.stdout:265: finishing write tid 1 to trial16941936-22 2026-02-25T08:42:58.186 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:42:58.189 INFO:tasks.rados.rados.0.trial169.stdout:266: finishing write tid 1 to trial16941936-34 2026-02-25T08:42:58.189 INFO:tasks.rados.rados.0.trial169.stdout:266: finishing write tid 2 to trial16941936-34 2026-02-25T08:42:58.192 INFO:tasks.rados.rados.0.trial169.stdout:266: finishing write tid 3 to trial16941936-34 2026-02-25T08:42:58.194 INFO:tasks.rados.rados.0.trial169.stdout:266: finishing write tid 4 to trial16941936-34 2026-02-25T08:42:58.194 INFO:tasks.rados.rados.0.trial169.stdout:266: oid 34 updating version 0 to 42 2026-02-25T08:42:58.194 INFO:tasks.rados.rados.0.trial169.stdout:266: oid 34 version 42 is already newer than 42 2026-02-25T08:42:58.194 INFO:tasks.rados.rados.0.trial169.stdout:266: oid 34 version 42 is already newer than 42 2026-02-25T08:42:58.194 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 34 v 42 (ObjNum 145 snap 32 seq_num 145) dirty exists 2026-02-25T08:42:58.194 INFO:tasks.rados.rados.0.trial169.stdout:266: left oid 34 (ObjNum 145 snap 32 seq_num 145) 2026-02-25T08:42:58.201 INFO:tasks.rados.rados.0.trial169.stdout:269: finishing write tid 1 to trial16941936-30 2026-02-25T08:42:58.207 INFO:tasks.rados.rados.0.trial169.stdout:265: finishing write tid 2 to trial16941936-22 2026-02-25T08:42:58.208 INFO:tasks.rados.rados.0.trial169.stdout:269: finishing write tid 2 to trial16941936-30 2026-02-25T08:42:58.215 INFO:tasks.rados.rados.0.trial169.stdout:269: finishing write tid 3 to trial16941936-30 2026-02-25T08:42:58.215 INFO:tasks.rados.rados.0.trial169.stdout:265: finishing write tid 3 to trial16941936-22 2026-02-25T08:42:58.218 INFO:tasks.rados.rados.0.trial169.stdout:265: finishing write tid 4 to trial16941936-22 2026-02-25T08:42:58.218 INFO:tasks.rados.rados.0.trial169.stdout:265: oid 22 updating version 0 to 22 2026-02-25T08:42:58.218 INFO:tasks.rados.rados.0.trial169.stdout:265: oid 22 updating version 22 to 24 2026-02-25T08:42:58.218 INFO:tasks.rados.rados.0.trial169.stdout:265: oid 22 version 24 is already newer than 24 2026-02-25T08:42:58.218 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 22 v 24 (ObjNum 144 snap 32 seq_num 144) dirty exists 2026-02-25T08:42:58.218 INFO:tasks.rados.rados.0.trial169.stdout:265: left oid 22 (ObjNum 144 snap 32 seq_num 144) 2026-02-25T08:42:58.219 INFO:tasks.rados.rados.0.trial169.stdout:269: finishing write tid 4 to trial16941936-30 2026-02-25T08:42:58.219 INFO:tasks.rados.rados.0.trial169.stdout:269: oid 30 updating version 0 to 51 2026-02-25T08:42:58.219 INFO:tasks.rados.rados.0.trial169.stdout:269: oid 30 version 51 is already newer than 51 2026-02-25T08:42:58.219 INFO:tasks.rados.rados.0.trial169.stdout:269: oid 30 version 51 is already newer than 51 2026-02-25T08:42:58.219 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 30 v 51 (ObjNum 147 snap 32 seq_num 147) dirty exists 2026-02-25T08:42:58.219 INFO:tasks.rados.rados.0.trial169.stdout:269: left oid 30 (ObjNum 147 snap 32 seq_num 147) 2026-02-25T08:42:58.230 INFO:tasks.rados.rados.0.trial169.stdout:265: done (6 left) 2026-02-25T08:42:58.231 INFO:tasks.rados.rados.0.trial169.stdout:266: done (5 left) 2026-02-25T08:42:58.231 INFO:tasks.rados.rados.0.trial169.stdout:268: done (4 left) 2026-02-25T08:42:58.231 INFO:tasks.rados.rados.0.trial169.stdout:269: done (3 left) 2026-02-25T08:42:58.231 INFO:tasks.rados.rados.0.trial169.stdout:270: done (2 left) 2026-02-25T08:42:58.231 INFO:tasks.rados.rados.0.trial169.stdout:271: write (excl) oid 8 current snap is 32 2026-02-25T08:42:58.231 INFO:tasks.rados.rados.0.trial169.stdout:271: seq_num 149 ranges {504243=622808,1819464=517046} 2026-02-25T08:42:58.234 INFO:tasks.rados.rados.0.trial169.stdout:271: writing trial16941936-8 from 504243 to 1127051 tid 1 2026-02-25T08:42:58.239 INFO:tasks.rados.rados.0.trial169.stdout:271: writing trial16941936-8 from 1819464 to 2336510 tid 2 2026-02-25T08:42:58.239 INFO:tasks.rados.rados.0.trial169.stdout:272: snap_create 2026-02-25T08:42:58.239 INFO:tasks.rados.rados.0.trial169.stdout:264: finishing write tid 1 to trial16941936-4 2026-02-25T08:42:58.242 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:42:58.250 INFO:tasks.rados.rados.0.trial169.stdout:271: finishing write tid 1 to trial16941936-8 2026-02-25T08:42:58.256 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:42:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:42:58.259 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:42:58.259 INFO:tasks.rados.rados.0.trial169.stdout:264: finishing write tid 2 to trial16941936-4 2026-02-25T08:42:58.265 INFO:tasks.rados.rados.0.trial169.stdout:271: finishing write tid 2 to trial16941936-8 2026-02-25T08:42:58.266 INFO:tasks.rados.rados.0.trial169.stdout:264: finishing write tid 3 to trial16941936-4 2026-02-25T08:42:58.268 INFO:tasks.rados.rados.0.trial169.stdout:264: finishing write tid 4 to trial16941936-4 2026-02-25T08:42:58.268 INFO:tasks.rados.rados.0.trial169.stdout:264: oid 4 updating version 0 to 112 2026-02-25T08:42:58.268 INFO:tasks.rados.rados.0.trial169.stdout:264: oid 4 version 112 is already newer than 112 2026-02-25T08:42:58.268 INFO:tasks.rados.rados.0.trial169.stdout:264: oid 4 version 112 is already newer than 110 2026-02-25T08:42:58.268 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 4 v 112 (ObjNum 143 snap 32 seq_num 143) dirty exists 2026-02-25T08:42:58.268 INFO:tasks.rados.rados.0.trial169.stdout:264: left oid 4 (ObjNum 143 snap 32 seq_num 143) 2026-02-25T08:42:58.269 INFO:tasks.rados.rados.0.trial169.stdout:271: finishing write tid 3 to trial16941936-8 2026-02-25T08:42:58.270 INFO:tasks.rados.rados.0.trial169.stdout:271: finishing write tid 4 to trial16941936-8 2026-02-25T08:42:58.270 INFO:tasks.rados.rados.0.trial169.stdout:271: oid 8 updating version 0 to 94 2026-02-25T08:42:58.270 INFO:tasks.rados.rados.0.trial169.stdout:271: oid 8 version 94 is already newer than 92 2026-02-25T08:42:58.270 INFO:tasks.rados.rados.0.trial169.stdout:271: oid 8 version 94 is already newer than 94 2026-02-25T08:42:58.271 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 8 v 94 (ObjNum 149 snap 32 seq_num 149) dirty exists 2026-02-25T08:42:58.271 INFO:tasks.rados.rados.0.trial169.stdout:271: left oid 8 (ObjNum 149 snap 32 seq_num 149) 2026-02-25T08:42:58.280 INFO:tasks.rados.rados.0.trial169.stdout:267: finishing write tid 1 to trial16941936-15 2026-02-25T08:42:58.286 INFO:tasks.rados.rados.0.trial169.stdout:267: finishing write tid 2 to trial16941936-15 2026-02-25T08:42:58.289 INFO:tasks.rados.rados.0.trial169.stdout:267: finishing write tid 3 to trial16941936-15 2026-02-25T08:42:58.290 INFO:tasks.rados.rados.0.trial169.stdout:267: finishing write tid 4 to trial16941936-15 2026-02-25T08:42:58.291 INFO:tasks.rados.rados.0.trial169.stdout:267: oid 15 updating version 0 to 28 2026-02-25T08:42:58.291 INFO:tasks.rados.rados.0.trial169.stdout:267: oid 15 version 28 is already newer than 28 2026-02-25T08:42:58.291 INFO:tasks.rados.rados.0.trial169.stdout:267: oid 15 version 28 is already newer than 28 2026-02-25T08:42:58.291 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 15 v 28 (ObjNum 146 snap 32 seq_num 146) dirty exists 2026-02-25T08:42:58.291 INFO:tasks.rados.rados.0.trial169.stdout:267: left oid 15 (ObjNum 146 snap 32 seq_num 146) 2026-02-25T08:42:58.319 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:58.316+0000 7f2b8e7c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:58.321 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:42:58.320+0000 7f2b8e7c1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:42:58.325 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:42:58.364 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:42:58.365 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:42:59.155 INFO:tasks.rados.rados.0.trial169.stdout:264: done (3 left) 2026-02-25T08:42:59.155 INFO:tasks.rados.rados.0.trial169.stdout:267: done (2 left) 2026-02-25T08:42:59.155 INFO:tasks.rados.rados.0.trial169.stdout:271: done (1 left) 2026-02-25T08:42:59.155 INFO:tasks.rados.rados.0.trial169.stdout:272: done (0 left) 2026-02-25T08:42:59.155 INFO:tasks.rados.rados.0.trial169.stdout:273: write oid 33 current snap is 33 2026-02-25T08:42:59.155 INFO:tasks.rados.rados.0.trial169.stdout:273: seq_num 150 ranges {521050=698057,1877946=643104} 2026-02-25T08:42:59.159 INFO:tasks.rados.rados.0.trial169.stdout:273: writing trial16941936-33 from 521050 to 1219107 tid 1 2026-02-25T08:42:59.164 INFO:tasks.rados.rados.0.trial169.stdout:273: writing trial16941936-33 from 1877946 to 2521050 tid 2 2026-02-25T08:42:59.164 INFO:tasks.rados.rados.0.trial169.stdout:274: read oid 5 snap -1 2026-02-25T08:42:59.165 INFO:tasks.rados.rados.0.trial169.stdout:274: expect deleted 2026-02-25T08:42:59.165 INFO:tasks.rados.rados.0.trial169.stdout:275: write oid 39 current snap is 33 2026-02-25T08:42:59.165 INFO:tasks.rados.rados.0.trial169.stdout:275: seq_num 151 ranges {537857=773306,1936428=520715} 2026-02-25T08:42:59.169 INFO:tasks.rados.rados.0.trial169.stdout:275: writing trial16941936-39 from 537857 to 1311163 tid 1 2026-02-25T08:42:59.174 INFO:tasks.rados.rados.0.trial169.stdout:275: writing trial16941936-39 from 1936428 to 2457143 tid 2 2026-02-25T08:42:59.174 INFO:tasks.rados.rados.0.trial169.stdout:276: read oid 37 snap -1 2026-02-25T08:42:59.174 INFO:tasks.rados.rados.0.trial169.stdout:276: expect (ObjNum 129 snap 23 seq_num 129) 2026-02-25T08:42:59.174 INFO:tasks.rados.rados.0.trial169.stdout:277: read oid 49 snap 24 2026-02-25T08:42:59.174 INFO:tasks.rados.rados.0.trial169.stdout:277: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:42:59.174 INFO:tasks.rados.rados.0.trial169.stdout:278: read oid 28 snap -1 2026-02-25T08:42:59.174 INFO:tasks.rados.rados.0.trial169.stdout:278: expect (ObjNum 135 snap 24 seq_num 135) 2026-02-25T08:42:59.174 INFO:tasks.rados.rados.0.trial169.stdout:279: read oid 4 snap 30 2026-02-25T08:42:59.174 INFO:tasks.rados.rados.0.trial169.stdout:279: expect (ObjNum 128 snap 23 seq_num 128) 2026-02-25T08:42:59.175 INFO:tasks.rados.rados.0.trial169.stdout:280: write (excl) oid 50 current snap is 33 2026-02-25T08:42:59.175 INFO:tasks.rados.rados.0.trial169.stdout:280: seq_num 152 ranges {554664=448555,1478557=664373} 2026-02-25T08:42:59.177 INFO:tasks.rados.rados.0.trial169.stdout:280: writing trial16941936-50 from 554664 to 1003219 tid 1 2026-02-25T08:42:59.182 INFO:tasks.rados.rados.0.trial169.stdout:280: writing trial16941936-50 from 1478557 to 2142930 tid 2 2026-02-25T08:42:59.182 INFO:tasks.rados.rados.0.trial169.stdout:281: snap_remove snap 32 2026-02-25T08:43:00.159 INFO:tasks.rados.rados.0.trial169.stdout:281: done (8 left) 2026-02-25T08:43:00.160 INFO:tasks.rados.rados.0.trial169.stdout:282: write oid 22 current snap is 33 2026-02-25T08:43:00.160 INFO:tasks.rados.rados.0.trial169.stdout:282: seq_num 153 ranges {571471=640157,1653392=524384} 2026-02-25T08:43:00.164 INFO:tasks.rados.rados.0.trial169.stdout:282: writing trial16941936-22 from 571471 to 1211628 tid 1 2026-02-25T08:43:00.168 INFO:tasks.rados.rados.0.trial169.stdout:282: writing trial16941936-22 from 1653392 to 2177776 tid 2 2026-02-25T08:43:00.168 INFO:tasks.rados.rados.0.trial169.stdout:283: read oid 3 snap 26 2026-02-25T08:43:00.168 INFO:tasks.rados.rados.0.trial169.stdout:283: expect deleted 2026-02-25T08:43:00.168 INFO:tasks.rados.rados.0.trial169.stdout:284: snap_create 2026-02-25T08:43:00.168 INFO:tasks.rados.rados.0.trial169.stdout:277: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:43:00.188 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:00.216 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 936733 2026-02-25T08:43:00.231 INFO:tasks.rados.rados.0.trial169.stdout:278: expect (ObjNum 135 snap 24 seq_num 135) 2026-02-25T08:43:00.277 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1908749 2026-02-25T08:43:00.280 INFO:tasks.rados.rados.0.trial169.stdout:276: expect (ObjNum 129 snap 23 seq_num 129) 2026-02-25T08:43:00.280 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 322587 2026-02-25T08:43:00.315 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 948152 2026-02-25T08:43:00.338 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:00.366 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:00.370 INFO:tasks.rados.rados.0.trial169.stdout:273: finishing write tid 1 to trial16941936-33 2026-02-25T08:43:00.370 INFO:tasks.rados.rados.0.trial169.stdout:273: finishing write tid 2 to trial16941936-33 2026-02-25T08:43:00.370 INFO:tasks.rados.rados.0.trial169.stdout:273: finishing write tid 3 to trial16941936-33 2026-02-25T08:43:00.371 INFO:tasks.rados.rados.0.trial169.stdout:275: finishing write tid 1 to trial16941936-39 2026-02-25T08:43:00.371 INFO:tasks.rados.rados.0.trial169.stdout:273: finishing write tid 4 to trial16941936-33 2026-02-25T08:43:00.371 INFO:tasks.rados.rados.0.trial169.stdout:273: oid 33 updating version 0 to 115 2026-02-25T08:43:00.371 INFO:tasks.rados.rados.0.trial169.stdout:273: oid 33 version 115 is already newer than 115 2026-02-25T08:43:00.371 INFO:tasks.rados.rados.0.trial169.stdout:273: oid 33 version 115 is already newer than 115 2026-02-25T08:43:00.371 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 33 v 115 (ObjNum 150 snap 33 seq_num 150) dirty exists 2026-02-25T08:43:00.371 INFO:tasks.rados.rados.0.trial169.stdout:273: left oid 33 (ObjNum 150 snap 33 seq_num 150) 2026-02-25T08:43:00.371 INFO:tasks.rados.rados.0.trial169.stdout:279: expect (ObjNum 128 snap 23 seq_num 128) 2026-02-25T08:43:00.385 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1758109 2026-02-25T08:43:00.395 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1936484 2026-02-25T08:43:00.402 INFO:tasks.rados.rados.0.trial169.stdout:275: finishing write tid 2 to trial16941936-39 2026-02-25T08:43:00.402 INFO:tasks.rados.rados.0.trial169.stdout:275: finishing write tid 3 to trial16941936-39 2026-02-25T08:43:00.402 INFO:tasks.rados.rados.0.trial169.stdout:275: finishing write tid 4 to trial16941936-39 2026-02-25T08:43:00.402 INFO:tasks.rados.rados.0.trial169.stdout:275: oid 39 updating version 0 to 57 2026-02-25T08:43:00.402 INFO:tasks.rados.rados.0.trial169.stdout:275: oid 39 version 57 is already newer than 57 2026-02-25T08:43:00.402 INFO:tasks.rados.rados.0.trial169.stdout:275: oid 39 version 57 is already newer than 57 2026-02-25T08:43:00.402 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 39 v 57 (ObjNum 151 snap 33 seq_num 151) dirty exists 2026-02-25T08:43:00.402 INFO:tasks.rados.rados.0.trial169.stdout:275: left oid 39 (ObjNum 151 snap 33 seq_num 151) 2026-02-25T08:43:00.402 INFO:tasks.rados.rados.0.trial169.stdout:280: finishing write tid 1 to trial16941936-50 2026-02-25T08:43:00.402 INFO:tasks.rados.rados.0.trial169.stdout:280: finishing write tid 2 to trial16941936-50 2026-02-25T08:43:00.402 INFO:tasks.rados.rados.0.trial169.stdout:280: finishing write tid 3 to trial16941936-50 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:280: finishing write tid 4 to trial16941936-50 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:280: oid 50 updating version 0 to 119 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:280: oid 50 version 119 is already newer than 117 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:280: oid 50 version 119 is already newer than 119 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 50 v 119 (ObjNum 152 snap 33 seq_num 152) dirty exists 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:280: left oid 50 (ObjNum 152 snap 33 seq_num 152) 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:282: finishing write tid 1 to trial16941936-22 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:282: finishing write tid 2 to trial16941936-22 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:282: finishing write tid 3 to trial16941936-22 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:282: finishing write tid 4 to trial16941936-22 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:282: oid 22 updating version 0 to 31 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:282: oid 22 version 31 is already newer than 31 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:282: oid 22 version 31 is already newer than 31 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 22 v 31 (ObjNum 153 snap 33 seq_num 153) dirty exists 2026-02-25T08:43:00.403 INFO:tasks.rados.rados.0.trial169.stdout:282: left oid 22 (ObjNum 153 snap 33 seq_num 153) 2026-02-25T08:43:00.441 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:00.438+0000 7faa7b162640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:00.443 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:00.442+0000 7faa7b162640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:00.447 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:00.487 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:01.057 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~55s 2026-02-25T08:43:01.162 INFO:tasks.rados.rados.0.trial169.stdout:273: done (10 left) 2026-02-25T08:43:01.162 INFO:tasks.rados.rados.0.trial169.stdout:274: done (9 left) 2026-02-25T08:43:01.162 INFO:tasks.rados.rados.0.trial169.stdout:275: done (8 left) 2026-02-25T08:43:01.162 INFO:tasks.rados.rados.0.trial169.stdout:276: done (7 left) 2026-02-25T08:43:01.162 INFO:tasks.rados.rados.0.trial169.stdout:277: done (6 left) 2026-02-25T08:43:01.162 INFO:tasks.rados.rados.0.trial169.stdout:278: done (5 left) 2026-02-25T08:43:01.162 INFO:tasks.rados.rados.0.trial169.stdout:279: done (4 left) 2026-02-25T08:43:01.162 INFO:tasks.rados.rados.0.trial169.stdout:280: done (3 left) 2026-02-25T08:43:01.162 INFO:tasks.rados.rados.0.trial169.stdout:282: done (2 left) 2026-02-25T08:43:01.163 INFO:tasks.rados.rados.0.trial169.stdout:283: done (1 left) 2026-02-25T08:43:01.163 INFO:tasks.rados.rados.0.trial169.stdout:284: done (0 left) 2026-02-25T08:43:01.163 INFO:tasks.rados.rados.0.trial169.stdout:285: snap_create 2026-02-25T08:43:02.166 INFO:tasks.rados.rados.0.trial169.stdout:285: done (0 left) 2026-02-25T08:43:02.167 INFO:tasks.rados.rados.0.trial169.stdout:286: delete oid 16 current snap is 35 2026-02-25T08:43:02.207 INFO:tasks.rados.rados.0.trial169.stdout:286: done (0 left) 2026-02-25T08:43:02.208 INFO:tasks.rados.rados.0.trial169.stdout:287: write oid 4 current snap is 35 2026-02-25T08:43:02.208 INFO:tasks.rados.rados.0.trial169.stdout:287: seq_num 155 ranges {605085=790655,2170356=434729} 2026-02-25T08:43:02.213 INFO:tasks.rados.rados.0.trial169.stdout:287: writing trial16941936-4 from 605085 to 1395740 tid 1 2026-02-25T08:43:02.218 INFO:tasks.rados.rados.0.trial169.stdout:287: writing trial16941936-4 from 2170356 to 2605085 tid 2 2026-02-25T08:43:02.218 INFO:tasks.rados.rados.0.trial169.stdout:288: read oid 29 snap -1 2026-02-25T08:43:02.218 INFO:tasks.rados.rados.0.trial169.stdout:288: expect (ObjNum 134 snap 24 seq_num 134) 2026-02-25T08:43:02.218 INFO:tasks.rados.rados.0.trial169.stdout:289: write oid 22 current snap is 35 2026-02-25T08:43:02.218 INFO:tasks.rados.rados.0.trial169.stdout:289: seq_num 156 ranges {621892=465904,1712485=671711} 2026-02-25T08:43:02.222 INFO:tasks.rados.rados.0.trial169.stdout:289: writing trial16941936-22 from 621892 to 1087796 tid 1 2026-02-25T08:43:02.227 INFO:tasks.rados.rados.0.trial169.stdout:289: writing trial16941936-22 from 1712485 to 2384196 tid 2 2026-02-25T08:43:02.227 INFO:tasks.rados.rados.0.trial169.stdout:290: write oid 36 current snap is 35 2026-02-25T08:43:02.227 INFO:tasks.rados.rados.0.trial169.stdout:290: seq_num 157 ranges {638699=541153,1770967=531722} 2026-02-25T08:43:02.231 INFO:tasks.rados.rados.0.trial169.stdout:290: writing trial16941936-36 from 638699 to 1179852 tid 1 2026-02-25T08:43:02.235 INFO:tasks.rados.rados.0.trial169.stdout:290: writing trial16941936-36 from 1770967 to 2302689 tid 2 2026-02-25T08:43:02.236 INFO:tasks.rados.rados.0.trial169.stdout:291: delete oid 19 current snap is 35 2026-02-25T08:43:02.247 INFO:tasks.rados.rados.0.trial169.stdout:288: expect (ObjNum 134 snap 24 seq_num 134) 2026-02-25T08:43:02.260 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:02.297 INFO:tasks.rados.rados.0.trial169.stdout:288: done (4 left) 2026-02-25T08:43:02.298 INFO:tasks.rados.rados.0.trial169.stdout:291: done (3 left) 2026-02-25T08:43:02.298 INFO:tasks.rados.rados.0.trial169.stdout:292: snap_remove snap 34 2026-02-25T08:43:02.488 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:02.567 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:02.564+0000 7fb7c85a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:02.569 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:02.568+0000 7fb7c85a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:02.573 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:02.611 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:02.611 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:03.167 INFO:tasks.rados.rados.0.trial169.stdout:292: done (3 left) 2026-02-25T08:43:03.168 INFO:tasks.rados.rados.0.trial169.stdout:293: read oid 43 snap -1 2026-02-25T08:43:03.168 INFO:tasks.rados.rados.0.trial169.stdout:293: expect (ObjNum 42 snap 0 seq_num 42) 2026-02-25T08:43:03.168 INFO:tasks.rados.rados.0.trial169.stdout:294: write (excl) oid 18 current snap is 35 2026-02-25T08:43:03.168 INFO:tasks.rados.rados.0.trial169.stdout:290: finishing write tid 1 to trial16941936-36 2026-02-25T08:43:03.168 INFO:tasks.rados.rados.0.trial169.stdout:294: seq_num 159 ranges {672313=691651,1887931=419038} 2026-02-25T08:43:03.172 INFO:tasks.rados.rados.0.trial169.stdout:294: writing trial16941936-18 from 672313 to 1363964 tid 1 2026-02-25T08:43:03.177 INFO:tasks.rados.rados.0.trial169.stdout:294: writing trial16941936-18 from 1887931 to 2306969 tid 2 2026-02-25T08:43:03.177 INFO:tasks.rados.rados.0.trial169.stdout:295: read oid 25 snap 33 2026-02-25T08:43:03.177 INFO:tasks.rados.rados.0.trial169.stdout:295: expect (ObjNum 78 snap 8 seq_num 78) 2026-02-25T08:43:03.177 INFO:tasks.rados.rados.0.trial169.stdout:296: snap_create 2026-02-25T08:43:03.177 INFO:tasks.rados.rados.0.trial169.stdout:287: finishing write tid 1 to trial16941936-4 2026-02-25T08:43:03.177 INFO:tasks.rados.rados.0.trial169.stdout:290: finishing write tid 2 to trial16941936-36 2026-02-25T08:43:03.177 INFO:tasks.rados.rados.0.trial169.stdout:290: finishing write tid 3 to trial16941936-36 2026-02-25T08:43:03.177 INFO:tasks.rados.rados.0.trial169.stdout:287: finishing write tid 2 to trial16941936-4 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:290: finishing write tid 4 to trial16941936-36 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:290: oid 36 updating version 0 to 60 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:290: oid 36 version 60 is already newer than 60 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:290: oid 36 version 60 is already newer than 60 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 36 v 60 (ObjNum 157 snap 35 seq_num 157) dirty exists 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:290: left oid 36 (ObjNum 157 snap 35 seq_num 157) 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:287: finishing write tid 3 to trial16941936-4 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:287: finishing write tid 4 to trial16941936-4 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:287: oid 4 updating version 0 to 125 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:287: oid 4 version 125 is already newer than 125 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:287: oid 4 version 125 is already newer than 125 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 4 v 125 (ObjNum 155 snap 35 seq_num 155) dirty exists 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:287: left oid 4 (ObjNum 155 snap 35 seq_num 155) 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:289: finishing write tid 1 to trial16941936-22 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:289: finishing write tid 2 to trial16941936-22 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:289: finishing write tid 3 to trial16941936-22 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:289: finishing write tid 4 to trial16941936-22 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:289: oid 22 updating version 0 to 37 2026-02-25T08:43:03.178 INFO:tasks.rados.rados.0.trial169.stdout:289: oid 22 version 37 is already newer than 37 2026-02-25T08:43:03.179 INFO:tasks.rados.rados.0.trial169.stdout:289: oid 22 version 37 is already newer than 37 2026-02-25T08:43:03.179 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 22 v 37 (ObjNum 156 snap 35 seq_num 156) dirty exists 2026-02-25T08:43:03.179 INFO:tasks.rados.rados.0.trial169.stdout:289: left oid 22 (ObjNum 156 snap 35 seq_num 156) 2026-02-25T08:43:03.212 INFO:tasks.rados.rados.0.trial169.stdout:293: expect (ObjNum 42 snap 0 seq_num 42) 2026-02-25T08:43:03.259 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:43:03.268 INFO:tasks.rados.rados.0.trial169.stdout:295: expect (ObjNum 78 snap 8 seq_num 78) 2026-02-25T08:43:03.268 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:03.307 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 97166 2026-02-25T08:43:03.329 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:03.333 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:43:03.345 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:03.384 INFO:tasks.rados.rados.0.trial169.stdout:294: finishing write tid 1 to trial16941936-18 2026-02-25T08:43:03.384 INFO:tasks.rados.rados.0.trial169.stdout:294: finishing write tid 2 to trial16941936-18 2026-02-25T08:43:03.385 INFO:tasks.rados.rados.0.trial169.stdout:294: finishing write tid 3 to trial16941936-18 2026-02-25T08:43:03.385 INFO:tasks.rados.rados.0.trial169.stdout:294: finishing write tid 4 to trial16941936-18 2026-02-25T08:43:03.385 INFO:tasks.rados.rados.0.trial169.stdout:294: oid 18 updating version 0 to 21 2026-02-25T08:43:03.385 INFO:tasks.rados.rados.0.trial169.stdout:294: oid 18 version 21 is already newer than 19 2026-02-25T08:43:03.385 INFO:tasks.rados.rados.0.trial169.stdout:294: oid 18 version 21 is already newer than 21 2026-02-25T08:43:03.385 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 18 v 21 (ObjNum 159 snap 35 seq_num 159) dirty exists 2026-02-25T08:43:03.385 INFO:tasks.rados.rados.0.trial169.stdout:294: left oid 18 (ObjNum 159 snap 35 seq_num 159) 2026-02-25T08:43:04.173 INFO:tasks.rados.rados.0.trial169.stdout:287: done (6 left) 2026-02-25T08:43:04.174 INFO:tasks.rados.rados.0.trial169.stdout:289: done (5 left) 2026-02-25T08:43:04.174 INFO:tasks.rados.rados.0.trial169.stdout:290: done (4 left) 2026-02-25T08:43:04.174 INFO:tasks.rados.rados.0.trial169.stdout:293: done (3 left) 2026-02-25T08:43:04.174 INFO:tasks.rados.rados.0.trial169.stdout:294: done (2 left) 2026-02-25T08:43:04.174 INFO:tasks.rados.rados.0.trial169.stdout:295: done (1 left) 2026-02-25T08:43:04.174 INFO:tasks.rados.rados.0.trial169.stdout:296: done (0 left) 2026-02-25T08:43:04.174 INFO:tasks.rados.rados.0.trial169.stdout:297: snap_remove snap 26 2026-02-25T08:43:04.613 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:04.692 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:04.689+0000 7fccf081a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:04.695 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:04.693+0000 7fccf081a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:04.698 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:04.740 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:04.741 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:05.175 INFO:tasks.rados.rados.0.trial169.stdout:297: done (0 left) 2026-02-25T08:43:05.176 INFO:tasks.rados.rados.0.trial169.stdout:298: delete oid 9 current snap is 36 2026-02-25T08:43:05.260 INFO:tasks.rados.rados.0.trial169.stdout:298: done (0 left) 2026-02-25T08:43:05.260 INFO:tasks.rados.rados.0.trial169.stdout:299: write oid 13 current snap is 36 2026-02-25T08:43:05.260 INFO:tasks.rados.rados.0.trial169.stdout:299: seq_num 161 ranges {705927=558502,2004895=422707} 2026-02-25T08:43:05.264 INFO:tasks.rados.rados.0.trial169.stdout:299: writing trial16941936-13 from 705927 to 1264429 tid 1 2026-02-25T08:43:05.269 INFO:tasks.rados.rados.0.trial169.stdout:299: writing trial16941936-13 from 2004895 to 2427602 tid 2 2026-02-25T08:43:05.269 INFO:tasks.rados.rados.0.trial169.stdout:300: write (excl) oid 14 current snap is 36 2026-02-25T08:43:05.269 INFO:tasks.rados.rados.0.trial169.stdout:300: seq_num 162 ranges {722734=633751,2063377=659357} 2026-02-25T08:43:05.274 INFO:tasks.rados.rados.0.trial169.stdout:300: writing trial16941936-14 from 722734 to 1356485 tid 1 2026-02-25T08:43:05.281 INFO:tasks.rados.rados.0.trial169.stdout:300: writing trial16941936-14 from 2063377 to 2722734 tid 2 2026-02-25T08:43:05.281 INFO:tasks.rados.rados.0.trial169.stdout:301: read oid 27 snap -1 2026-02-25T08:43:05.281 INFO:tasks.rados.rados.0.trial169.stdout:301: expect deleted 2026-02-25T08:43:05.281 INFO:tasks.rados.rados.0.trial169.stdout:302: snap_create 2026-02-25T08:43:05.288 INFO:tasks.rados.rados.0.trial169.stdout:300: finishing write tid 1 to trial16941936-14 2026-02-25T08:43:05.300 INFO:tasks.rados.rados.0.trial169.stdout:299: finishing write tid 1 to trial16941936-13 2026-02-25T08:43:05.302 INFO:tasks.rados.rados.0.trial169.stdout:300: finishing write tid 2 to trial16941936-14 2026-02-25T08:43:05.304 INFO:tasks.rados.rados.0.trial169.stdout:299: finishing write tid 2 to trial16941936-13 2026-02-25T08:43:05.308 INFO:tasks.rados.rados.0.trial169.stdout:299: finishing write tid 3 to trial16941936-13 2026-02-25T08:43:05.308 INFO:tasks.rados.rados.0.trial169.stdout:300: finishing write tid 3 to trial16941936-14 2026-02-25T08:43:05.310 INFO:tasks.rados.rados.0.trial169.stdout:299: finishing write tid 4 to trial16941936-13 2026-02-25T08:43:05.310 INFO:tasks.rados.rados.0.trial169.stdout:299: oid 13 updating version 0 to 100 2026-02-25T08:43:05.310 INFO:tasks.rados.rados.0.trial169.stdout:299: oid 13 version 100 is already newer than 100 2026-02-25T08:43:05.310 INFO:tasks.rados.rados.0.trial169.stdout:299: oid 13 version 100 is already newer than 100 2026-02-25T08:43:05.310 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 13 v 100 (ObjNum 161 snap 36 seq_num 161) dirty exists 2026-02-25T08:43:05.310 INFO:tasks.rados.rados.0.trial169.stdout:299: left oid 13 (ObjNum 161 snap 36 seq_num 161) 2026-02-25T08:43:05.311 INFO:tasks.rados.rados.0.trial169.stdout:300: finishing write tid 4 to trial16941936-14 2026-02-25T08:43:05.311 INFO:tasks.rados.rados.0.trial169.stdout:300: oid 14 updating version 0 to 71 2026-02-25T08:43:05.311 INFO:tasks.rados.rados.0.trial169.stdout:300: oid 14 version 71 is already newer than 69 2026-02-25T08:43:05.311 INFO:tasks.rados.rados.0.trial169.stdout:300: oid 14 version 71 is already newer than 71 2026-02-25T08:43:05.311 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 14 v 71 (ObjNum 162 snap 36 seq_num 162) dirty exists 2026-02-25T08:43:05.311 INFO:tasks.rados.rados.0.trial169.stdout:300: left oid 14 (ObjNum 162 snap 36 seq_num 162) 2026-02-25T08:43:06.180 INFO:tasks.rados.rados.0.trial169.stdout:299: done (3 left) 2026-02-25T08:43:06.181 INFO:tasks.rados.rados.0.trial169.stdout:300: done (2 left) 2026-02-25T08:43:06.181 INFO:tasks.rados.rados.0.trial169.stdout:301: done (1 left) 2026-02-25T08:43:06.181 INFO:tasks.rados.rados.0.trial169.stdout:302: done (0 left) 2026-02-25T08:43:06.181 INFO:tasks.rados.rados.0.trial169.stdout:303: snap_remove snap 24 2026-02-25T08:43:06.560 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~61s 2026-02-25T08:43:06.742 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:06.821 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:06.818+0000 7fa8f9a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:06.823 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:06.822+0000 7fa8f9a5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:06.827 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:06.869 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:06.870 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:07.183 INFO:tasks.rados.rados.0.trial169.stdout:303: done (0 left) 2026-02-25T08:43:07.184 INFO:tasks.rados.rados.0.trial169.stdout:304: read oid 22 snap -1 2026-02-25T08:43:07.184 INFO:tasks.rados.rados.0.trial169.stdout:304: expect (ObjNum 156 snap 35 seq_num 156) 2026-02-25T08:43:07.184 INFO:tasks.rados.rados.0.trial169.stdout:305: write oid 10 current snap is 37 2026-02-25T08:43:07.184 INFO:tasks.rados.rados.0.trial169.stdout:305: seq_num 163 ranges {739541=709000,2121859=426376} 2026-02-25T08:43:07.189 INFO:tasks.rados.rados.0.trial169.stdout:305: writing trial16941936-10 from 739541 to 1448541 tid 1 2026-02-25T08:43:07.193 INFO:tasks.rados.rados.0.trial169.stdout:305: writing trial16941936-10 from 2121859 to 2548235 tid 2 2026-02-25T08:43:07.193 INFO:tasks.rados.rados.0.trial169.stdout:306: delete oid 15 current snap is 37 2026-02-25T08:43:07.227 INFO:tasks.rados.rados.0.trial169.stdout:305: finishing write tid 1 to trial16941936-10 2026-02-25T08:43:07.251 INFO:tasks.rados.rados.0.trial169.stdout:305: finishing write tid 2 to trial16941936-10 2026-02-25T08:43:07.285 INFO:tasks.rados.rados.0.trial169.stdout:305: finishing write tid 3 to trial16941936-10 2026-02-25T08:43:07.285 INFO:tasks.rados.rados.0.trial169.stdout:304: expect (ObjNum 156 snap 35 seq_num 156) 2026-02-25T08:43:07.311 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2257210 2026-02-25T08:43:07.325 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 367434 2026-02-25T08:43:07.383 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1416698 2026-02-25T08:43:07.414 INFO:tasks.rados.rados.0.trial169.stdout:305: finishing write tid 4 to trial16941936-10 2026-02-25T08:43:07.414 INFO:tasks.rados.rados.0.trial169.stdout:305: oid 10 updating version 0 to 17 2026-02-25T08:43:07.414 INFO:tasks.rados.rados.0.trial169.stdout:305: oid 10 version 17 is already newer than 17 2026-02-25T08:43:07.414 INFO:tasks.rados.rados.0.trial169.stdout:305: oid 10 version 17 is already newer than 17 2026-02-25T08:43:07.414 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 10 v 17 (ObjNum 163 snap 37 seq_num 163) dirty exists 2026-02-25T08:43:07.414 INFO:tasks.rados.rados.0.trial169.stdout:305: left oid 10 (ObjNum 163 snap 37 seq_num 163) 2026-02-25T08:43:07.414 INFO:tasks.rados.rados.0.trial169.stdout:304: done (2 left) 2026-02-25T08:43:07.414 INFO:tasks.rados.rados.0.trial169.stdout:305: done (1 left) 2026-02-25T08:43:07.414 INFO:tasks.rados.rados.0.trial169.stdout:306: done (0 left) 2026-02-25T08:43:07.415 INFO:tasks.rados.rados.0.trial169.stdout:307: snap_create 2026-02-25T08:43:08.124 INFO:tasks.rados.rados.0.trial169.stdout:307: done (0 left) 2026-02-25T08:43:08.124 INFO:tasks.rados.rados.0.trial169.stdout:308: write (excl) oid 13 current snap is 38 2026-02-25T08:43:08.124 INFO:tasks.rados.rados.0.trial169.stdout:308: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2026-02-25T08:43:08.130 INFO:tasks.rados.rados.0.trial169.stdout:308: writing trial16941936-13 from 773155 to 1232653 tid 1 2026-02-25T08:43:08.133 INFO:tasks.rados.rados.0.trial169.stdout:308: writing trial16941936-13 from 1722470 to 2152515 tid 2 2026-02-25T08:43:08.136 INFO:tasks.rados.rados.0.trial169.stdout:308: writing trial16941936-13 from 2648676 to 2773155 tid 3 2026-02-25T08:43:08.136 INFO:tasks.rados.rados.0.trial169.stdout:309: write oid 48 current snap is 38 2026-02-25T08:43:08.136 INFO:tasks.rados.rados.0.trial169.stdout:309: seq_num 166 ranges {789962=534747,1780952=690056} 2026-02-25T08:43:08.141 INFO:tasks.rados.rados.0.trial169.stdout:309: writing trial16941936-48 from 789962 to 1324709 tid 1 2026-02-25T08:43:08.145 INFO:tasks.rados.rados.0.trial169.stdout:309: writing trial16941936-48 from 1780952 to 2471008 tid 2 2026-02-25T08:43:08.145 INFO:tasks.rados.rados.0.trial169.stdout:310: write (excl) oid 12 current snap is 38 2026-02-25T08:43:08.145 INFO:tasks.rados.rados.0.trial169.stdout:310: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2026-02-25T08:43:08.149 INFO:tasks.rados.rados.0.trial169.stdout:310: writing trial16941936-12 from 406769 to 1016765 tid 1 2026-02-25T08:43:08.152 INFO:tasks.rados.rados.0.trial169.stdout:310: writing trial16941936-12 from 1439434 to 1873148 tid 2 2026-02-25T08:43:08.156 INFO:tasks.rados.rados.0.trial169.stdout:310: writing trial16941936-12 from 2303522 to 2806769 tid 3 2026-02-25T08:43:08.156 INFO:tasks.rados.rados.0.trial169.stdout:311: snap_remove snap 36 2026-02-25T08:43:08.333 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:43:08.405 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:08.408 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:08.408 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:43:08.871 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:08.947 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:08.944+0000 7f867e35c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:08.949 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:08.948+0000 7f867e35c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:08.953 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:08.993 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:08.994 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:09.126 INFO:tasks.rados.rados.0.trial169.stdout:311: done (3 left) 2026-02-25T08:43:09.126 INFO:tasks.rados.rados.0.trial169.stdout:312: snap_create 2026-02-25T08:43:09.126 INFO:tasks.rados.rados.0.trial169.stdout:309: finishing write tid 1 to trial16941936-48 2026-02-25T08:43:09.126 INFO:tasks.rados.rados.0.trial169.stdout:309: finishing write tid 2 to trial16941936-48 2026-02-25T08:43:09.126 INFO:tasks.rados.rados.0.trial169.stdout:308: finishing write tid 1 to trial16941936-13 2026-02-25T08:43:09.126 INFO:tasks.rados.rados.0.trial169.stdout:309: finishing write tid 3 to trial16941936-48 2026-02-25T08:43:09.127 INFO:tasks.rados.rados.0.trial169.stdout:309: finishing write tid 4 to trial16941936-48 2026-02-25T08:43:09.127 INFO:tasks.rados.rados.0.trial169.stdout:309: oid 48 updating version 0 to 10 2026-02-25T08:43:09.127 INFO:tasks.rados.rados.0.trial169.stdout:309: oid 48 version 10 is already newer than 10 2026-02-25T08:43:09.127 INFO:tasks.rados.rados.0.trial169.stdout:309: oid 48 version 10 is already newer than 10 2026-02-25T08:43:09.127 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 48 v 10 (ObjNum 166 snap 38 seq_num 166) dirty exists 2026-02-25T08:43:09.127 INFO:tasks.rados.rados.0.trial169.stdout:309: left oid 48 (ObjNum 166 snap 38 seq_num 166) 2026-02-25T08:43:09.127 INFO:tasks.rados.rados.0.trial169.stdout:310: finishing write tid 1 to trial16941936-12 2026-02-25T08:43:09.127 INFO:tasks.rados.rados.0.trial169.stdout:308: finishing write tid 2 to trial16941936-13 2026-02-25T08:43:09.127 INFO:tasks.rados.rados.0.trial169.stdout:308: finishing write tid 3 to trial16941936-13 2026-02-25T08:43:09.127 INFO:tasks.rados.rados.0.trial169.stdout:308: finishing write tid 4 to trial16941936-13 2026-02-25T08:43:09.127 INFO:tasks.rados.rados.0.trial169.stdout:310: finishing write tid 2 to trial16941936-12 2026-02-25T08:43:09.128 INFO:tasks.rados.rados.0.trial169.stdout:308: finishing write tid 5 to trial16941936-13 2026-02-25T08:43:09.128 INFO:tasks.rados.rados.0.trial169.stdout:308: oid 13 updating version 0 to 106 2026-02-25T08:43:09.128 INFO:tasks.rados.rados.0.trial169.stdout:308: oid 13 updating version 106 to 109 2026-02-25T08:43:09.128 INFO:tasks.rados.rados.0.trial169.stdout:308: oid 13 version 109 is already newer than 109 2026-02-25T08:43:09.128 INFO:tasks.rados.rados.0.trial169.stdout:308: oid 13 version 109 is already newer than 109 2026-02-25T08:43:09.128 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 13 v 109 (ObjNum 165 snap 38 seq_num 165) dirty exists 2026-02-25T08:43:09.128 INFO:tasks.rados.rados.0.trial169.stdout:308: left oid 13 (ObjNum 165 snap 38 seq_num 165) 2026-02-25T08:43:09.128 INFO:tasks.rados.rados.0.trial169.stdout:310: finishing write tid 3 to trial16941936-12 2026-02-25T08:43:09.128 INFO:tasks.rados.rados.0.trial169.stdout:310: finishing write tid 4 to trial16941936-12 2026-02-25T08:43:09.128 INFO:tasks.rados.rados.0.trial169.stdout:310: finishing write tid 5 to trial16941936-12 2026-02-25T08:43:09.128 INFO:tasks.rados.rados.0.trial169.stdout:310: oid 12 updating version 0 to 39 2026-02-25T08:43:09.128 INFO:tasks.rados.rados.0.trial169.stdout:310: oid 12 version 39 is already newer than 39 2026-02-25T08:43:09.128 INFO:tasks.rados.rados.0.trial169.stdout:310: oid 12 version 39 is already newer than 36 2026-02-25T08:43:09.129 INFO:tasks.rados.rados.0.trial169.stdout:310: oid 12 version 39 is already newer than 39 2026-02-25T08:43:09.129 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 12 v 39 (ObjNum 167 snap 38 seq_num 167) dirty exists 2026-02-25T08:43:09.129 INFO:tasks.rados.rados.0.trial169.stdout:310: left oid 12 (ObjNum 167 snap 38 seq_num 167) 2026-02-25T08:43:10.126 INFO:tasks.rados.rados.0.trial169.stdout:308: done (3 left) 2026-02-25T08:43:10.126 INFO:tasks.rados.rados.0.trial169.stdout:309: done (2 left) 2026-02-25T08:43:10.126 INFO:tasks.rados.rados.0.trial169.stdout:310: done (1 left) 2026-02-25T08:43:10.126 INFO:tasks.rados.rados.0.trial169.stdout:312: done (0 left) 2026-02-25T08:43:10.127 INFO:tasks.rados.rados.0.trial169.stdout:313: snap_create 2026-02-25T08:43:10.995 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:11.073 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:11.070+0000 7fba69c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:11.075 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:11.073+0000 7fba69c5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:11.078 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:11.117 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:11.117 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:11.125 INFO:tasks.rados.rados.0.trial169.stdout:313: done (0 left) 2026-02-25T08:43:11.125 INFO:tasks.rados.rados.0.trial169.stdout:314: write (excl) oid 14 current snap is 40 2026-02-25T08:43:11.125 INFO:tasks.rados.rados.0.trial169.stdout:314: seq_num 168 ranges {423576=401598,1497916=693725} 2026-02-25T08:43:11.128 INFO:tasks.rados.rados.0.trial169.stdout:314: writing trial16941936-14 from 423576 to 825174 tid 1 2026-02-25T08:43:11.134 INFO:tasks.rados.rados.0.trial169.stdout:314: writing trial16941936-14 from 1497916 to 2191641 tid 2 2026-02-25T08:43:11.134 INFO:tasks.rados.rados.0.trial169.stdout:315: write (excl) oid 17 current snap is 40 2026-02-25T08:43:11.134 INFO:tasks.rados.rados.0.trial169.stdout:315: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2026-02-25T08:43:11.137 INFO:tasks.rados.rados.0.trial169.stdout:315: writing trial16941936-17 from 440383 to 917230 tid 1 2026-02-25T08:43:11.141 INFO:tasks.rados.rados.0.trial169.stdout:315: writing trial16941936-17 from 1556398 to 1993781 tid 2 2026-02-25T08:43:11.144 INFO:tasks.rados.rados.0.trial169.stdout:315: writing trial16941936-17 from 2474721 to 2840383 tid 3 2026-02-25T08:43:11.144 INFO:tasks.rados.rados.0.trial169.stdout:316: write oid 24 current snap is 40 2026-02-25T08:43:11.145 INFO:tasks.rados.rados.0.trial169.stdout:316: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2026-02-25T08:43:11.148 INFO:tasks.rados.rados.0.trial169.stdout:316: writing trial16941936-24 from 457190 to 1009286 tid 1 2026-02-25T08:43:11.152 INFO:tasks.rados.rados.0.trial169.stdout:316: writing trial16941936-24 from 1614880 to 2195921 tid 2 2026-02-25T08:43:11.155 INFO:tasks.rados.rados.0.trial169.stdout:316: writing trial16941936-24 from 2785791 to 2857190 tid 3 2026-02-25T08:43:11.155 INFO:tasks.rados.rados.0.trial169.stdout:317: snap_remove snap 19 2026-02-25T08:43:12.064 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~66s 2026-02-25T08:43:12.132 INFO:tasks.rados.rados.0.trial169.stdout:317: done (3 left) 2026-02-25T08:43:12.132 INFO:tasks.rados.rados.0.trial169.stdout:318: snap_create 2026-02-25T08:43:12.132 INFO:tasks.rados.rados.0.trial169.stdout:314: finishing write tid 1 to trial16941936-14 2026-02-25T08:43:12.132 INFO:tasks.rados.rados.0.trial169.stdout:315: finishing write tid 1 to trial16941936-17 2026-02-25T08:43:12.132 INFO:tasks.rados.rados.0.trial169.stdout:314: finishing write tid 2 to trial16941936-14 2026-02-25T08:43:12.132 INFO:tasks.rados.rados.0.trial169.stdout:314: finishing write tid 3 to trial16941936-14 2026-02-25T08:43:12.132 INFO:tasks.rados.rados.0.trial169.stdout:315: finishing write tid 2 to trial16941936-17 2026-02-25T08:43:12.133 INFO:tasks.rados.rados.0.trial169.stdout:314: finishing write tid 4 to trial16941936-14 2026-02-25T08:43:12.133 INFO:tasks.rados.rados.0.trial169.stdout:314: oid 14 updating version 0 to 75 2026-02-25T08:43:12.133 INFO:tasks.rados.rados.0.trial169.stdout:314: oid 14 version 75 is already newer than 75 2026-02-25T08:43:12.133 INFO:tasks.rados.rados.0.trial169.stdout:314: oid 14 version 75 is already newer than 73 2026-02-25T08:43:12.133 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 14 v 75 (ObjNum 168 snap 40 seq_num 168) dirty exists 2026-02-25T08:43:12.133 INFO:tasks.rados.rados.0.trial169.stdout:314: left oid 14 (ObjNum 168 snap 40 seq_num 168) 2026-02-25T08:43:12.133 INFO:tasks.rados.rados.0.trial169.stdout:315: finishing write tid 3 to trial16941936-17 2026-02-25T08:43:12.133 INFO:tasks.rados.rados.0.trial169.stdout:315: finishing write tid 4 to trial16941936-17 2026-02-25T08:43:12.133 INFO:tasks.rados.rados.0.trial169.stdout:315: finishing write tid 5 to trial16941936-17 2026-02-25T08:43:12.133 INFO:tasks.rados.rados.0.trial169.stdout:315: oid 17 updating version 0 to 32 2026-02-25T08:43:12.133 INFO:tasks.rados.rados.0.trial169.stdout:315: oid 17 updating version 32 to 35 2026-02-25T08:43:12.133 INFO:tasks.rados.rados.0.trial169.stdout:315: oid 17 version 35 is already newer than 35 2026-02-25T08:43:12.134 INFO:tasks.rados.rados.0.trial169.stdout:315: oid 17 version 35 is already newer than 35 2026-02-25T08:43:12.134 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 17 v 35 (ObjNum 169 snap 40 seq_num 169) dirty exists 2026-02-25T08:43:12.134 INFO:tasks.rados.rados.0.trial169.stdout:315: left oid 17 (ObjNum 169 snap 40 seq_num 169) 2026-02-25T08:43:12.134 INFO:tasks.rados.rados.0.trial169.stdout:316: finishing write tid 1 to trial16941936-24 2026-02-25T08:43:12.134 INFO:tasks.rados.rados.0.trial169.stdout:316: finishing write tid 2 to trial16941936-24 2026-02-25T08:43:12.134 INFO:tasks.rados.rados.0.trial169.stdout:316: finishing write tid 3 to trial16941936-24 2026-02-25T08:43:12.134 INFO:tasks.rados.rados.0.trial169.stdout:316: finishing write tid 4 to trial16941936-24 2026-02-25T08:43:12.134 INFO:tasks.rados.rados.0.trial169.stdout:316: finishing write tid 5 to trial16941936-24 2026-02-25T08:43:12.134 INFO:tasks.rados.rados.0.trial169.stdout:316: oid 24 updating version 0 to 44 2026-02-25T08:43:12.134 INFO:tasks.rados.rados.0.trial169.stdout:316: oid 24 version 44 is already newer than 44 2026-02-25T08:43:12.134 INFO:tasks.rados.rados.0.trial169.stdout:316: oid 24 version 44 is already newer than 44 2026-02-25T08:43:12.134 INFO:tasks.rados.rados.0.trial169.stdout:316: oid 24 version 44 is already newer than 44 2026-02-25T08:43:12.134 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 24 v 44 (ObjNum 170 snap 40 seq_num 170) dirty exists 2026-02-25T08:43:12.135 INFO:tasks.rados.rados.0.trial169.stdout:316: left oid 24 (ObjNum 170 snap 40 seq_num 170) 2026-02-25T08:43:13.119 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:13.131 INFO:tasks.rados.rados.0.trial169.stdout:314: done (3 left) 2026-02-25T08:43:13.131 INFO:tasks.rados.rados.0.trial169.stdout:315: done (2 left) 2026-02-25T08:43:13.131 INFO:tasks.rados.rados.0.trial169.stdout:316: done (1 left) 2026-02-25T08:43:13.131 INFO:tasks.rados.rados.0.trial169.stdout:318: done (0 left) 2026-02-25T08:43:13.132 INFO:tasks.rados.rados.0.trial169.stdout:319: delete oid 12 current snap is 41 2026-02-25T08:43:13.173 INFO:tasks.rados.rados.0.trial169.stdout:319: done (0 left) 2026-02-25T08:43:13.173 INFO:tasks.rados.rados.0.trial169.stdout:320: write (excl) oid 43 current snap is 41 2026-02-25T08:43:13.174 INFO:tasks.rados.rados.0.trial169.stdout:320: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2026-02-25T08:43:13.179 INFO:tasks.rados.rados.0.trial169.stdout:320: writing trial16941936-43 from 490804 to 1193398 tid 1 2026-02-25T08:43:13.182 INFO:tasks.rados.rados.0.trial169.stdout:320: writing trial16941936-43 from 1615491 to 2200201 tid 2 2026-02-25T08:43:13.185 INFO:tasks.rados.rados.0.trial169.stdout:320: writing trial16941936-43 from 2724284 to 2890804 tid 3 2026-02-25T08:43:13.185 INFO:tasks.rados.rados.0.trial169.stdout:321: snap_create 2026-02-25T08:43:13.196 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:13.194+0000 7f3f0ac5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:13.199 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:13.197+0000 7f3f0ac5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:13.203 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:13.207 INFO:tasks.rados.rados.0.trial169.stdout:320: finishing write tid 1 to trial16941936-43 2026-02-25T08:43:13.220 INFO:tasks.rados.rados.0.trial169.stdout:320: finishing write tid 2 to trial16941936-43 2026-02-25T08:43:13.228 INFO:tasks.rados.rados.0.trial169.stdout:320: finishing write tid 3 to trial16941936-43 2026-02-25T08:43:13.228 INFO:tasks.rados.rados.0.trial169.stdout:320: finishing write tid 4 to trial16941936-43 2026-02-25T08:43:13.230 INFO:tasks.rados.rados.0.trial169.stdout:320: finishing write tid 5 to trial16941936-43 2026-02-25T08:43:13.230 INFO:tasks.rados.rados.0.trial169.stdout:320: oid 43 updating version 0 to 24 2026-02-25T08:43:13.231 INFO:tasks.rados.rados.0.trial169.stdout:320: oid 43 version 24 is already newer than 21 2026-02-25T08:43:13.231 INFO:tasks.rados.rados.0.trial169.stdout:320: oid 43 version 24 is already newer than 24 2026-02-25T08:43:13.231 INFO:tasks.rados.rados.0.trial169.stdout:320: oid 43 version 24 is already newer than 24 2026-02-25T08:43:13.231 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 43 v 24 (ObjNum 172 snap 41 seq_num 172) dirty exists 2026-02-25T08:43:13.231 INFO:tasks.rados.rados.0.trial169.stdout:320: left oid 43 (ObjNum 172 snap 41 seq_num 172) 2026-02-25T08:43:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:13.243 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:13.410 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:43:13.479 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:13.481 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:13.482 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:43:14.137 INFO:tasks.rados.rados.0.trial169.stdout:320: done (1 left) 2026-02-25T08:43:14.137 INFO:tasks.rados.rados.0.trial169.stdout:321: done (0 left) 2026-02-25T08:43:14.137 INFO:tasks.rados.rados.0.trial169.stdout:322: snap_create 2026-02-25T08:43:15.139 INFO:tasks.rados.rados.0.trial169.stdout:322: done (0 left) 2026-02-25T08:43:15.139 INFO:tasks.rados.rados.0.trial169.stdout:323: write oid 45 current snap is 43 2026-02-25T08:43:15.140 INFO:tasks.rados.rados.0.trial169.stdout:323: seq_num 173 ranges {507611=777843,2073973=444721} 2026-02-25T08:43:15.144 INFO:tasks.rados.rados.0.trial169.stdout:323: writing trial16941936-45 from 507611 to 1285454 tid 1 2026-02-25T08:43:15.149 INFO:tasks.rados.rados.0.trial169.stdout:323: writing trial16941936-45 from 2073973 to 2518694 tid 2 2026-02-25T08:43:15.149 INFO:tasks.rados.rados.0.trial169.stdout:324: snap_create 2026-02-25T08:43:15.187 INFO:tasks.rados.rados.0.trial169.stdout:323: finishing write tid 1 to trial16941936-45 2026-02-25T08:43:15.192 INFO:tasks.rados.rados.0.trial169.stdout:323: finishing write tid 2 to trial16941936-45 2026-02-25T08:43:15.195 INFO:tasks.rados.rados.0.trial169.stdout:323: finishing write tid 3 to trial16941936-45 2026-02-25T08:43:15.197 INFO:tasks.rados.rados.0.trial169.stdout:323: finishing write tid 4 to trial16941936-45 2026-02-25T08:43:15.197 INFO:tasks.rados.rados.0.trial169.stdout:323: oid 45 updating version 0 to 79 2026-02-25T08:43:15.197 INFO:tasks.rados.rados.0.trial169.stdout:323: oid 45 version 79 is already newer than 79 2026-02-25T08:43:15.197 INFO:tasks.rados.rados.0.trial169.stdout:323: oid 45 version 79 is already newer than 79 2026-02-25T08:43:15.197 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 45 v 79 (ObjNum 173 snap 43 seq_num 173) dirty exists 2026-02-25T08:43:15.197 INFO:tasks.rados.rados.0.trial169.stdout:323: left oid 45 (ObjNum 173 snap 43 seq_num 173) 2026-02-25T08:43:15.245 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:15.322 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:15.319+0000 7fd404ab7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:15.324 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:15.322+0000 7fd404ab7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:15.327 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:15.373 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:15.374 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:16.145 INFO:tasks.rados.rados.0.trial169.stdout:323: done (1 left) 2026-02-25T08:43:16.145 INFO:tasks.rados.rados.0.trial169.stdout:324: done (0 left) 2026-02-25T08:43:16.145 INFO:tasks.rados.rados.0.trial169.stdout:325: snap_create 2026-02-25T08:43:17.147 INFO:tasks.rados.rados.0.trial169.stdout:325: done (0 left) 2026-02-25T08:43:17.147 INFO:tasks.rados.rados.0.trial169.stdout:326: delete oid 13 current snap is 45 2026-02-25T08:43:17.192 INFO:tasks.rados.rados.0.trial169.stdout:326: done (0 left) 2026-02-25T08:43:17.192 INFO:tasks.rados.rados.0.trial169.stdout:327: write (excl) oid 36 current snap is 45 2026-02-25T08:43:17.192 INFO:tasks.rados.rados.0.trial169.stdout:327: seq_num 175 ranges {541225=644694,1907290=448390} 2026-02-25T08:43:17.196 INFO:tasks.rados.rados.0.trial169.stdout:327: writing trial16941936-36 from 541225 to 1185919 tid 1 2026-02-25T08:43:17.201 INFO:tasks.rados.rados.0.trial169.stdout:327: writing trial16941936-36 from 1907290 to 2355680 tid 2 2026-02-25T08:43:17.201 INFO:tasks.rados.rados.0.trial169.stdout:328: snap_create 2026-02-25T08:43:17.223 INFO:tasks.rados.rados.0.trial169.stdout:327: finishing write tid 1 to trial16941936-36 2026-02-25T08:43:17.235 INFO:tasks.rados.rados.0.trial169.stdout:327: finishing write tid 2 to trial16941936-36 2026-02-25T08:43:17.237 INFO:tasks.rados.rados.0.trial169.stdout:327: finishing write tid 3 to trial16941936-36 2026-02-25T08:43:17.239 INFO:tasks.rados.rados.0.trial169.stdout:327: finishing write tid 4 to trial16941936-36 2026-02-25T08:43:17.239 INFO:tasks.rados.rados.0.trial169.stdout:327: oid 36 updating version 0 to 62 2026-02-25T08:43:17.239 INFO:tasks.rados.rados.0.trial169.stdout:327: oid 36 updating version 62 to 64 2026-02-25T08:43:17.239 INFO:tasks.rados.rados.0.trial169.stdout:327: oid 36 version 64 is already newer than 64 2026-02-25T08:43:17.239 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 36 v 64 (ObjNum 175 snap 45 seq_num 175) dirty exists 2026-02-25T08:43:17.239 INFO:tasks.rados.rados.0.trial169.stdout:327: left oid 36 (ObjNum 175 snap 45 seq_num 175) 2026-02-25T08:43:17.375 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:17.453 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:17.450+0000 7efc0905a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:17.455 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:17.454+0000 7efc0905a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:17.459 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:17.499 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:17.500 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:17.569 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~72s 2026-02-25T08:43:18.125 INFO:tasks.rados.rados.0.trial169.stdout:327: done (1 left) 2026-02-25T08:43:18.125 INFO:tasks.rados.rados.0.trial169.stdout:328: done (0 left) 2026-02-25T08:43:18.125 INFO:tasks.rados.rados.0.trial169.stdout:329: read oid 48 snap -1 2026-02-25T08:43:18.125 INFO:tasks.rados.rados.0.trial169.stdout:329: expect (ObjNum 166 snap 38 seq_num 166) 2026-02-25T08:43:18.125 INFO:tasks.rados.rados.0.trial169.stdout:330: snap_remove snap 43 2026-02-25T08:43:18.483 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:43:18.555 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:18.557 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:18.558 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:43:19.127 INFO:tasks.rados.rados.0.trial169.stdout:330: done (1 left) 2026-02-25T08:43:19.127 INFO:tasks.rados.rados.0.trial169.stdout:331: write oid 27 current snap is 46 2026-02-25T08:43:19.127 INFO:tasks.rados.rados.0.trial169.stdout:331: seq_num 176 ranges {558032=719943,1849419=592048} 2026-02-25T08:43:19.131 INFO:tasks.rados.rados.0.trial169.stdout:331: writing trial16941936-27 from 558032 to 1277975 tid 1 2026-02-25T08:43:19.136 INFO:tasks.rados.rados.0.trial169.stdout:331: writing trial16941936-27 from 1849419 to 2441467 tid 2 2026-02-25T08:43:19.136 INFO:tasks.rados.rados.0.trial169.stdout:332: read oid 23 snap -1 2026-02-25T08:43:19.137 INFO:tasks.rados.rados.0.trial169.stdout:332: expect (ObjNum 22 snap 0 seq_num 22) 2026-02-25T08:43:19.137 INFO:tasks.rados.rados.0.trial169.stdout:333: write oid 43 current snap is 46 2026-02-25T08:43:19.137 INFO:tasks.rados.rados.0.trial169.stdout:329: expect (ObjNum 166 snap 38 seq_num 166) 2026-02-25T08:43:19.143 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2512409 2026-02-25T08:43:19.174 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:19.198 INFO:tasks.rados.rados.0.trial169.stdout:332: expect (ObjNum 22 snap 0 seq_num 22) 2026-02-25T08:43:19.198 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:19.253 INFO:tasks.rados.rados.0.trial169.stdout:331: finishing write tid 1 to trial16941936-27 2026-02-25T08:43:19.253 INFO:tasks.rados.rados.0.trial169.stdout:331: finishing write tid 2 to trial16941936-27 2026-02-25T08:43:19.253 INFO:tasks.rados.rados.0.trial169.stdout:331: finishing write tid 3 to trial16941936-27 2026-02-25T08:43:19.253 INFO:tasks.rados.rados.0.trial169.stdout:331: finishing write tid 4 to trial16941936-27 2026-02-25T08:43:19.253 INFO:tasks.rados.rados.0.trial169.stdout:331: oid 27 updating version 0 to 27 2026-02-25T08:43:19.253 INFO:tasks.rados.rados.0.trial169.stdout:331: oid 27 version 27 is already newer than 27 2026-02-25T08:43:19.253 INFO:tasks.rados.rados.0.trial169.stdout:331: oid 27 version 27 is already newer than 27 2026-02-25T08:43:19.253 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 27 v 27 (ObjNum 176 snap 46 seq_num 176) dirty exists 2026-02-25T08:43:19.253 INFO:tasks.rados.rados.0.trial169.stdout:331: left oid 27 (ObjNum 176 snap 46 seq_num 176) 2026-02-25T08:43:19.254 INFO:tasks.rados.rados.0.trial169.stdout:333: seq_num 177 ranges {574839=795192,1907901=452059} 2026-02-25T08:43:19.258 INFO:tasks.rados.rados.0.trial169.stdout:333: writing trial16941936-43 from 574839 to 1370031 tid 1 2026-02-25T08:43:19.262 INFO:tasks.rados.rados.0.trial169.stdout:333: writing trial16941936-43 from 1907901 to 2359960 tid 2 2026-02-25T08:43:19.262 INFO:tasks.rados.rados.0.trial169.stdout:329: done (3 left) 2026-02-25T08:43:19.262 INFO:tasks.rados.rados.0.trial169.stdout:331: done (2 left) 2026-02-25T08:43:19.262 INFO:tasks.rados.rados.0.trial169.stdout:332: done (1 left) 2026-02-25T08:43:19.263 INFO:tasks.rados.rados.0.trial169.stdout:334: write oid 28 current snap is 46 2026-02-25T08:43:19.263 INFO:tasks.rados.rados.0.trial169.stdout:334: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2026-02-25T08:43:19.266 INFO:tasks.rados.rados.0.trial169.stdout:334: writing trial16941936-28 from 591646 to 1062087 tid 1 2026-02-25T08:43:19.270 INFO:tasks.rados.rados.0.trial169.stdout:334: writing trial16941936-28 from 1566383 to 2162100 tid 2 2026-02-25T08:43:19.274 INFO:tasks.rados.rados.0.trial169.stdout:334: writing trial16941936-28 from 2888822 to 2991646 tid 3 2026-02-25T08:43:19.274 INFO:tasks.rados.rados.0.trial169.stdout:335: read oid 15 snap -1 2026-02-25T08:43:19.274 INFO:tasks.rados.rados.0.trial169.stdout:335: expect deleted 2026-02-25T08:43:19.274 INFO:tasks.rados.rados.0.trial169.stdout:336: write (excl) oid 7 current snap is 46 2026-02-25T08:43:19.274 INFO:tasks.rados.rados.0.trial169.stdout:336: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2026-02-25T08:43:19.278 INFO:tasks.rados.rados.0.trial169.stdout:336: writing trial16941936-7 from 608453 to 1154143 tid 1 2026-02-25T08:43:19.281 INFO:tasks.rados.rados.0.trial169.stdout:336: writing trial16941936-7 from 1624865 to 2080593 tid 2 2026-02-25T08:43:19.285 INFO:tasks.rados.rados.0.trial169.stdout:336: writing trial16941936-7 from 2632598 to 3008453 tid 3 2026-02-25T08:43:19.285 INFO:tasks.rados.rados.0.trial169.stdout:337: read oid 32 snap 38 2026-02-25T08:43:19.285 INFO:tasks.rados.rados.0.trial169.stdout:337: expect (ObjNum 127 snap 21 seq_num 127) 2026-02-25T08:43:19.285 INFO:tasks.rados.rados.0.trial169.stdout:338: read oid 50 snap 41 2026-02-25T08:43:19.285 INFO:tasks.rados.rados.0.trial169.stdout:338: expect (ObjNum 152 snap 33 seq_num 152) 2026-02-25T08:43:19.286 INFO:tasks.rados.rados.0.trial169.stdout:339: write oid 48 current snap is 46 2026-02-25T08:43:19.286 INFO:tasks.rados.rados.0.trial169.stdout:339: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2026-02-25T08:43:19.289 INFO:tasks.rados.rados.0.trial169.stdout:339: writing trial16941936-48 from 625260 to 1246199 tid 1 2026-02-25T08:43:19.293 INFO:tasks.rados.rados.0.trial169.stdout:339: writing trial16941936-48 from 1683347 to 2282733 tid 2 2026-02-25T08:43:19.297 INFO:tasks.rados.rados.0.trial169.stdout:339: writing trial16941936-48 from 2943668 to 3025260 tid 3 2026-02-25T08:43:19.297 INFO:tasks.rados.rados.0.trial169.stdout:340: read oid 23 snap -1 2026-02-25T08:43:19.297 INFO:tasks.rados.rados.0.trial169.stdout:340: expect (ObjNum 22 snap 0 seq_num 22) 2026-02-25T08:43:19.297 INFO:tasks.rados.rados.0.trial169.stdout:341: delete oid 20 current snap is 46 2026-02-25T08:43:19.297 INFO:tasks.rados.rados.0.trial169.stdout:333: finishing write tid 1 to trial16941936-43 2026-02-25T08:43:19.306 INFO:tasks.rados.rados.0.trial169.stdout:333: finishing write tid 2 to trial16941936-43 2026-02-25T08:43:19.308 INFO:tasks.rados.rados.0.trial169.stdout:333: finishing write tid 3 to trial16941936-43 2026-02-25T08:43:19.309 INFO:tasks.rados.rados.0.trial169.stdout:337: expect (ObjNum 127 snap 21 seq_num 127) 2026-02-25T08:43:19.309 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:19.329 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 624563 2026-02-25T08:43:19.341 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:19.359 INFO:tasks.rados.rados.0.trial169.stdout:333: finishing write tid 4 to trial16941936-43 2026-02-25T08:43:19.359 INFO:tasks.rados.rados.0.trial169.stdout:333: oid 43 updating version 0 to 31 2026-02-25T08:43:19.359 INFO:tasks.rados.rados.0.trial169.stdout:333: oid 43 version 31 is already newer than 31 2026-02-25T08:43:19.359 INFO:tasks.rados.rados.0.trial169.stdout:333: oid 43 version 31 is already newer than 31 2026-02-25T08:43:19.359 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 43 v 31 (ObjNum 177 snap 46 seq_num 177) dirty exists 2026-02-25T08:43:19.359 INFO:tasks.rados.rados.0.trial169.stdout:333: left oid 43 (ObjNum 177 snap 46 seq_num 177) 2026-02-25T08:43:19.359 INFO:tasks.rados.rados.0.trial169.stdout:334: finishing write tid 1 to trial16941936-28 2026-02-25T08:43:19.359 INFO:tasks.rados.rados.0.trial169.stdout:334: finishing write tid 2 to trial16941936-28 2026-02-25T08:43:19.359 INFO:tasks.rados.rados.0.trial169.stdout:340: expect (ObjNum 22 snap 0 seq_num 22) 2026-02-25T08:43:19.362 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 863304 2026-02-25T08:43:19.405 INFO:tasks.rados.rados.0.trial169.stdout:336: finishing write tid 1 to trial16941936-7 2026-02-25T08:43:19.406 INFO:tasks.rados.rados.0.trial169.stdout:334: finishing write tid 3 to trial16941936-28 2026-02-25T08:43:19.406 INFO:tasks.rados.rados.0.trial169.stdout:333: done (8 left) 2026-02-25T08:43:19.406 INFO:tasks.rados.rados.0.trial169.stdout:335: done (7 left) 2026-02-25T08:43:19.406 INFO:tasks.rados.rados.0.trial169.stdout:337: done (6 left) 2026-02-25T08:43:19.406 INFO:tasks.rados.rados.0.trial169.stdout:340: done (5 left) 2026-02-25T08:43:19.406 INFO:tasks.rados.rados.0.trial169.stdout:341: done (4 left) 2026-02-25T08:43:19.406 INFO:tasks.rados.rados.0.trial169.stdout:342: write oid 36 current snap is 46 2026-02-25T08:43:19.406 INFO:tasks.rados.rados.0.trial169.stdout:334: finishing write tid 4 to trial16941936-28 2026-02-25T08:43:19.407 INFO:tasks.rados.rados.0.trial169.stdout:334: finishing write tid 5 to trial16941936-28 2026-02-25T08:43:19.407 INFO:tasks.rados.rados.0.trial169.stdout:334: oid 28 updating version 0 to 34 2026-02-25T08:43:19.407 INFO:tasks.rados.rados.0.trial169.stdout:334: oid 28 version 34 is already newer than 34 2026-02-25T08:43:19.407 INFO:tasks.rados.rados.0.trial169.stdout:334: oid 28 version 34 is already newer than 34 2026-02-25T08:43:19.407 INFO:tasks.rados.rados.0.trial169.stdout:334: oid 28 version 34 is already newer than 34 2026-02-25T08:43:19.407 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 28 v 34 (ObjNum 178 snap 46 seq_num 178) dirty exists 2026-02-25T08:43:19.407 INFO:tasks.rados.rados.0.trial169.stdout:334: left oid 28 (ObjNum 178 snap 46 seq_num 178) 2026-02-25T08:43:19.407 INFO:tasks.rados.rados.0.trial169.stdout:336: finishing write tid 2 to trial16941936-7 2026-02-25T08:43:19.407 INFO:tasks.rados.rados.0.trial169.stdout:336: finishing write tid 3 to trial16941936-7 2026-02-25T08:43:19.407 INFO:tasks.rados.rados.0.trial169.stdout:336: finishing write tid 4 to trial16941936-7 2026-02-25T08:43:19.407 INFO:tasks.rados.rados.0.trial169.stdout:336: finishing write tid 5 to trial16941936-7 2026-02-25T08:43:19.408 INFO:tasks.rados.rados.0.trial169.stdout:336: oid 7 updating version 0 to 147 2026-02-25T08:43:19.408 INFO:tasks.rados.rados.0.trial169.stdout:336: oid 7 updating version 147 to 150 2026-02-25T08:43:19.408 INFO:tasks.rados.rados.0.trial169.stdout:336: oid 7 version 150 is already newer than 150 2026-02-25T08:43:19.408 INFO:tasks.rados.rados.0.trial169.stdout:336: oid 7 version 150 is already newer than 150 2026-02-25T08:43:19.408 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 7 v 150 (ObjNum 179 snap 46 seq_num 179) dirty exists 2026-02-25T08:43:19.408 INFO:tasks.rados.rados.0.trial169.stdout:336: left oid 7 (ObjNum 179 snap 46 seq_num 179) 2026-02-25T08:43:19.408 INFO:tasks.rados.rados.0.trial169.stdout:338: expect (ObjNum 152 snap 33 seq_num 152) 2026-02-25T08:43:19.416 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1659925 2026-02-25T08:43:19.438 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:19.500 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:19.513 INFO:tasks.rados.rados.0.trial169.stdout:339: finishing write tid 1 to trial16941936-48 2026-02-25T08:43:19.513 INFO:tasks.rados.rados.0.trial169.stdout:342: seq_num 182 ranges {658874=771437,2083958=603055} 2026-02-25T08:43:19.518 INFO:tasks.rados.rados.0.trial169.stdout:342: writing trial16941936-36 from 658874 to 1430311 tid 1 2026-02-25T08:43:19.523 INFO:tasks.rados.rados.0.trial169.stdout:342: writing trial16941936-36 from 2083958 to 2687013 tid 2 2026-02-25T08:43:19.524 INFO:tasks.rados.rados.0.trial169.stdout:334: done (4 left) 2026-02-25T08:43:19.524 INFO:tasks.rados.rados.0.trial169.stdout:336: done (3 left) 2026-02-25T08:43:19.524 INFO:tasks.rados.rados.0.trial169.stdout:338: done (2 left) 2026-02-25T08:43:19.524 INFO:tasks.rados.rados.0.trial169.stdout:343: read oid 19 snap -1 2026-02-25T08:43:19.524 INFO:tasks.rados.rados.0.trial169.stdout:343: expect deleted 2026-02-25T08:43:19.524 INFO:tasks.rados.rados.0.trial169.stdout:344: write (excl) oid 40 current snap is 46 2026-02-25T08:43:19.524 INFO:tasks.rados.rados.0.trial169.stdout:339: finishing write tid 2 to trial16941936-48 2026-02-25T08:43:19.524 INFO:tasks.rados.rados.0.trial169.stdout:339: finishing write tid 3 to trial16941936-48 2026-02-25T08:43:19.524 INFO:tasks.rados.rados.0.trial169.stdout:339: finishing write tid 4 to trial16941936-48 2026-02-25T08:43:19.524 INFO:tasks.rados.rados.0.trial169.stdout:339: finishing write tid 5 to trial16941936-48 2026-02-25T08:43:19.524 INFO:tasks.rados.rados.0.trial169.stdout:339: oid 48 updating version 0 to 15 2026-02-25T08:43:19.524 INFO:tasks.rados.rados.0.trial169.stdout:339: oid 48 version 15 is already newer than 15 2026-02-25T08:43:19.525 INFO:tasks.rados.rados.0.trial169.stdout:339: oid 48 version 15 is already newer than 15 2026-02-25T08:43:19.525 INFO:tasks.rados.rados.0.trial169.stdout:339: oid 48 version 15 is already newer than 15 2026-02-25T08:43:19.525 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 48 v 15 (ObjNum 180 snap 46 seq_num 180) dirty exists 2026-02-25T08:43:19.525 INFO:tasks.rados.rados.0.trial169.stdout:339: left oid 48 (ObjNum 180 snap 46 seq_num 180) 2026-02-25T08:43:19.525 INFO:tasks.rados.rados.0.trial169.stdout:344: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2026-02-25T08:43:19.527 INFO:tasks.rados.rados.0.trial169.stdout:344: writing trial16941936-40 from 675681 to 1238720 tid 1 2026-02-25T08:43:19.533 INFO:tasks.rados.rados.0.trial169.stdout:344: writing trial16941936-40 from 1858793 to 2605506 tid 2 2026-02-25T08:43:19.535 INFO:tasks.rados.rados.0.trial169.stdout:344: writing trial16941936-40 from 3025937 to 3075681 tid 3 2026-02-25T08:43:19.535 INFO:tasks.rados.rados.0.trial169.stdout:339: done (3 left) 2026-02-25T08:43:19.535 INFO:tasks.rados.rados.0.trial169.stdout:345: snap_remove snap 33 2026-02-25T08:43:19.574 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:19.572+0000 7f7b4139c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:19.578 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:19.576+0000 7f7b4139c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:19.581 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:19.624 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:19.625 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:20.129 INFO:tasks.rados.rados.0.trial169.stdout:345: done (3 left) 2026-02-25T08:43:20.130 INFO:tasks.rados.rados.0.trial169.stdout:346: snap_remove snap 41 2026-02-25T08:43:21.135 INFO:tasks.rados.rados.0.trial169.stdout:346: done (3 left) 2026-02-25T08:43:21.135 INFO:tasks.rados.rados.0.trial169.stdout:347: delete oid 31 current snap is 46 2026-02-25T08:43:21.135 INFO:tasks.rados.rados.0.trial169.stdout:342: finishing write tid 1 to trial16941936-36 2026-02-25T08:43:21.135 INFO:tasks.rados.rados.0.trial169.stdout:342: finishing write tid 2 to trial16941936-36 2026-02-25T08:43:21.135 INFO:tasks.rados.rados.0.trial169.stdout:342: finishing write tid 3 to trial16941936-36 2026-02-25T08:43:21.136 INFO:tasks.rados.rados.0.trial169.stdout:344: finishing write tid 1 to trial16941936-40 2026-02-25T08:43:21.136 INFO:tasks.rados.rados.0.trial169.stdout:342: finishing write tid 4 to trial16941936-36 2026-02-25T08:43:21.136 INFO:tasks.rados.rados.0.trial169.stdout:342: oid 36 updating version 0 to 70 2026-02-25T08:43:21.136 INFO:tasks.rados.rados.0.trial169.stdout:342: oid 36 version 70 is already newer than 70 2026-02-25T08:43:21.136 INFO:tasks.rados.rados.0.trial169.stdout:342: oid 36 version 70 is already newer than 70 2026-02-25T08:43:21.136 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 36 v 70 (ObjNum 182 snap 46 seq_num 182) dirty exists 2026-02-25T08:43:21.136 INFO:tasks.rados.rados.0.trial169.stdout:342: left oid 36 (ObjNum 182 snap 46 seq_num 182) 2026-02-25T08:43:21.136 INFO:tasks.rados.rados.0.trial169.stdout:344: finishing write tid 2 to trial16941936-40 2026-02-25T08:43:21.136 INFO:tasks.rados.rados.0.trial169.stdout:344: finishing write tid 3 to trial16941936-40 2026-02-25T08:43:21.136 INFO:tasks.rados.rados.0.trial169.stdout:344: finishing write tid 4 to trial16941936-40 2026-02-25T08:43:21.136 INFO:tasks.rados.rados.0.trial169.stdout:344: finishing write tid 5 to trial16941936-40 2026-02-25T08:43:21.136 INFO:tasks.rados.rados.0.trial169.stdout:344: oid 40 updating version 0 to 155 2026-02-25T08:43:21.137 INFO:tasks.rados.rados.0.trial169.stdout:344: oid 40 version 155 is already newer than 152 2026-02-25T08:43:21.137 INFO:tasks.rados.rados.0.trial169.stdout:344: oid 40 version 155 is already newer than 155 2026-02-25T08:43:21.137 INFO:tasks.rados.rados.0.trial169.stdout:344: oid 40 version 155 is already newer than 155 2026-02-25T08:43:21.137 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 40 v 155 (ObjNum 183 snap 46 seq_num 183) dirty exists 2026-02-25T08:43:21.137 INFO:tasks.rados.rados.0.trial169.stdout:344: left oid 40 (ObjNum 183 snap 46 seq_num 183) 2026-02-25T08:43:21.175 INFO:tasks.rados.rados.0.trial169.stdout:342: done (3 left) 2026-02-25T08:43:21.175 INFO:tasks.rados.rados.0.trial169.stdout:343: done (2 left) 2026-02-25T08:43:21.175 INFO:tasks.rados.rados.0.trial169.stdout:344: done (1 left) 2026-02-25T08:43:21.175 INFO:tasks.rados.rados.0.trial169.stdout:347: done (0 left) 2026-02-25T08:43:21.175 INFO:tasks.rados.rados.0.trial169.stdout:348: read oid 15 snap 40 2026-02-25T08:43:21.175 INFO:tasks.rados.rados.0.trial169.stdout:348: expect deleted 2026-02-25T08:43:21.175 INFO:tasks.rados.rados.0.trial169.stdout:349: snap_remove snap 37 2026-02-25T08:43:21.626 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:21.707 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:21.704+0000 7f3a318cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:21.709 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:21.708+0000 7f3a318cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:21.713 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:21.752 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:22.138 INFO:tasks.rados.rados.0.trial169.stdout:349: done (1 left) 2026-02-25T08:43:22.138 INFO:tasks.rados.rados.0.trial169.stdout:350: snap_create 2026-02-25T08:43:23.074 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~77s 2026-02-25T08:43:23.139 INFO:tasks.rados.rados.0.trial169.stdout:348: done (1 left) 2026-02-25T08:43:23.139 INFO:tasks.rados.rados.0.trial169.stdout:350: done (0 left) 2026-02-25T08:43:23.139 INFO:tasks.rados.rados.0.trial169.stdout:351: snap_remove snap 30 2026-02-25T08:43:23.559 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:43:23.631 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:23.634 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:23.635 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:43:23.753 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:23.831 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:23.828+0000 7f461c45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:23.834 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:23.832+0000 7f461c45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:23.837 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:23.878 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:23.879 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:24.143 INFO:tasks.rados.rados.0.trial169.stdout:351: done (0 left) 2026-02-25T08:43:24.143 INFO:tasks.rados.rados.0.trial169.stdout:352: read oid 6 snap -1 2026-02-25T08:43:24.143 INFO:tasks.rados.rados.0.trial169.stdout:352: expect (ObjNum 57 snap 1 seq_num 57) 2026-02-25T08:43:24.143 INFO:tasks.rados.rados.0.trial169.stdout:353: snap_remove snap 42 2026-02-25T08:43:25.149 INFO:tasks.rados.rados.0.trial169.stdout:353: done (1 left) 2026-02-25T08:43:25.149 INFO:tasks.rados.rados.0.trial169.stdout:354: read oid 38 snap -1 2026-02-25T08:43:25.149 INFO:tasks.rados.rados.0.trial169.stdout:354: expect deleted 2026-02-25T08:43:25.149 INFO:tasks.rados.rados.0.trial169.stdout:355: snap_create 2026-02-25T08:43:25.149 INFO:tasks.rados.rados.0.trial169.stdout:352: expect (ObjNum 57 snap 1 seq_num 57) 2026-02-25T08:43:25.184 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:25.880 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:25.958 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:25.955+0000 7f08b27cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:25.961 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:25.959+0000 7f08b27cc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:25.965 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:26.005 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:26.006 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:26.151 INFO:tasks.rados.rados.0.trial169.stdout:352: done (2 left) 2026-02-25T08:43:26.151 INFO:tasks.rados.rados.0.trial169.stdout:354: done (1 left) 2026-02-25T08:43:26.151 INFO:tasks.rados.rados.0.trial169.stdout:355: done (0 left) 2026-02-25T08:43:26.151 INFO:tasks.rados.rados.0.trial169.stdout:356: snap_remove snap 25 2026-02-25T08:43:27.155 INFO:tasks.rados.rados.0.trial169.stdout:356: done (0 left) 2026-02-25T08:43:27.156 INFO:tasks.rados.rados.0.trial169.stdout:357: snap_create 2026-02-25T08:43:28.007 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:28.085 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:28.083+0000 7f96eb6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:28.088 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:28.086+0000 7f96eb6cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:28.091 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:28.126 INFO:tasks.rados.rados.0.trial169.stdout:357: done (0 left) 2026-02-25T08:43:28.126 INFO:tasks.rados.rados.0.trial169.stdout:358: snap_remove snap 35 2026-02-25T08:43:28.130 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:28.131 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:28.577 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~83s 2026-02-25T08:43:28.636 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:43:28.705 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:28.708 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:28.708 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:43:29.128 INFO:tasks.rados.rados.0.trial169.stdout:358: done (0 left) 2026-02-25T08:43:29.128 INFO:tasks.rados.rados.0.trial169.stdout:359: delete oid 48 current snap is 49 2026-02-25T08:43:29.166 INFO:tasks.rados.rados.0.trial169.stdout:359: done (0 left) 2026-02-25T08:43:29.166 INFO:tasks.rados.rados.0.trial169.stdout:360: read oid 46 snap -1 2026-02-25T08:43:29.166 INFO:tasks.rados.rados.0.trial169.stdout:360: expect deleted 2026-02-25T08:43:29.166 INFO:tasks.rados.rados.0.trial169.stdout:361: write (excl) oid 44 current snap is 49 2026-02-25T08:43:29.166 INFO:tasks.rados.rados.0.trial169.stdout:361: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2026-02-25T08:43:29.172 INFO:tasks.rados.rados.0.trial169.stdout:361: writing trial16941936-44 from 726102 to 1514888 tid 1 2026-02-25T08:43:29.176 INFO:tasks.rados.rados.0.trial169.stdout:361: writing trial16941936-44 from 1917886 to 2528279 tid 2 2026-02-25T08:43:29.179 INFO:tasks.rados.rados.0.trial169.stdout:361: writing trial16941936-44 from 3108206 to 3126102 tid 3 2026-02-25T08:43:29.179 INFO:tasks.rados.rados.0.trial169.stdout:362: write oid 11 current snap is 49 2026-02-25T08:43:29.179 INFO:tasks.rados.rados.0.trial169.stdout:362: seq_num 187 ranges {742909=464035,1976368=754051} 2026-02-25T08:43:29.183 INFO:tasks.rados.rados.0.trial169.stdout:362: writing trial16941936-11 from 742909 to 1206944 tid 1 2026-02-25T08:43:29.189 INFO:tasks.rados.rados.0.trial169.stdout:362: writing trial16941936-11 from 1976368 to 2730419 tid 2 2026-02-25T08:43:29.189 INFO:tasks.rados.rados.0.trial169.stdout:363: snap_create 2026-02-25T08:43:29.189 INFO:tasks.rados.rados.0.trial169.stdout:361: finishing write tid 1 to trial16941936-44 2026-02-25T08:43:29.201 INFO:tasks.rados.rados.0.trial169.stdout:361: finishing write tid 2 to trial16941936-44 2026-02-25T08:43:29.206 INFO:tasks.rados.rados.0.trial169.stdout:361: finishing write tid 3 to trial16941936-44 2026-02-25T08:43:29.209 INFO:tasks.rados.rados.0.trial169.stdout:361: finishing write tid 4 to trial16941936-44 2026-02-25T08:43:29.211 INFO:tasks.rados.rados.0.trial169.stdout:361: finishing write tid 5 to trial16941936-44 2026-02-25T08:43:29.211 INFO:tasks.rados.rados.0.trial169.stdout:361: oid 44 updating version 0 to 74 2026-02-25T08:43:29.211 INFO:tasks.rados.rados.0.trial169.stdout:361: oid 44 version 74 is already newer than 74 2026-02-25T08:43:29.211 INFO:tasks.rados.rados.0.trial169.stdout:361: oid 44 version 74 is already newer than 74 2026-02-25T08:43:29.211 INFO:tasks.rados.rados.0.trial169.stdout:361: oid 44 version 74 is already newer than 71 2026-02-25T08:43:29.211 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 44 v 74 (ObjNum 186 snap 49 seq_num 186) dirty exists 2026-02-25T08:43:29.211 INFO:tasks.rados.rados.0.trial169.stdout:361: left oid 44 (ObjNum 186 snap 49 seq_num 186) 2026-02-25T08:43:29.214 INFO:tasks.rados.rados.0.trial169.stdout:362: finishing write tid 1 to trial16941936-11 2026-02-25T08:43:29.219 INFO:tasks.rados.rados.0.trial169.stdout:362: finishing write tid 2 to trial16941936-11 2026-02-25T08:43:29.225 INFO:tasks.rados.rados.0.trial169.stdout:362: finishing write tid 3 to trial16941936-11 2026-02-25T08:43:29.226 INFO:tasks.rados.rados.0.trial169.stdout:362: finishing write tid 4 to trial16941936-11 2026-02-25T08:43:29.226 INFO:tasks.rados.rados.0.trial169.stdout:362: oid 11 updating version 0 to 38 2026-02-25T08:43:29.226 INFO:tasks.rados.rados.0.trial169.stdout:362: oid 11 version 38 is already newer than 38 2026-02-25T08:43:29.226 INFO:tasks.rados.rados.0.trial169.stdout:362: oid 11 version 38 is already newer than 38 2026-02-25T08:43:29.226 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 11 v 38 (ObjNum 187 snap 49 seq_num 187) dirty exists 2026-02-25T08:43:29.226 INFO:tasks.rados.rados.0.trial169.stdout:362: left oid 11 (ObjNum 187 snap 49 seq_num 187) 2026-02-25T08:43:30.130 INFO:tasks.rados.rados.0.trial169.stdout:360: done (3 left) 2026-02-25T08:43:30.130 INFO:tasks.rados.rados.0.trial169.stdout:361: done (2 left) 2026-02-25T08:43:30.130 INFO:tasks.rados.rados.0.trial169.stdout:362: done (1 left) 2026-02-25T08:43:30.131 INFO:tasks.rados.rados.0.trial169.stdout:363: done (0 left) 2026-02-25T08:43:30.131 INFO:tasks.rados.rados.0.trial169.stdout:364: snap_remove snap 29 2026-02-25T08:43:30.132 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:30.207 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:30.205+0000 7f27b8f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:30.210 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:30.208+0000 7f27b8f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:30.213 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:30.251 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:30.251 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:31.133 INFO:tasks.rados.rados.0.trial169.stdout:364: done (0 left) 2026-02-25T08:43:31.133 INFO:tasks.rados.rados.0.trial169.stdout:365: snap_create 2026-02-25T08:43:32.140 INFO:tasks.rados.rados.0.trial169.stdout:365: done (0 left) 2026-02-25T08:43:32.140 INFO:tasks.rados.rados.0.trial169.stdout:366: read oid 32 snap -1 2026-02-25T08:43:32.140 INFO:tasks.rados.rados.0.trial169.stdout:366: expect (ObjNum 127 snap 21 seq_num 127) 2026-02-25T08:43:32.140 INFO:tasks.rados.rados.0.trial169.stdout:367: snap_remove snap 31 2026-02-25T08:43:32.252 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:32.327 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:32.324+0000 7f3d5ad0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:32.330 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:32.328+0000 7f3d5ad0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:32.333 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:32.392 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:32.393 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:33.142 INFO:tasks.rados.rados.0.trial169.stdout:367: done (1 left) 2026-02-25T08:43:33.142 INFO:tasks.rados.rados.0.trial169.stdout:368: snap_create 2026-02-25T08:43:33.142 INFO:tasks.rados.rados.0.trial169.stdout:366: expect (ObjNum 127 snap 21 seq_num 127) 2026-02-25T08:43:33.142 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:33.160 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:33.179 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:33.709 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:43:33.780 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:33.782 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:33.783 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:43:34.081 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~88s 2026-02-25T08:43:34.147 INFO:tasks.rados.rados.0.trial169.stdout:366: done (1 left) 2026-02-25T08:43:34.147 INFO:tasks.rados.rados.0.trial169.stdout:368: done (0 left) 2026-02-25T08:43:34.147 INFO:tasks.rados.rados.0.trial169.stdout:369: read oid 5 snap -1 2026-02-25T08:43:34.147 INFO:tasks.rados.rados.0.trial169.stdout:369: expect deleted 2026-02-25T08:43:34.148 INFO:tasks.rados.rados.0.trial169.stdout:370: snap_create 2026-02-25T08:43:34.394 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:34.472 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:34.469+0000 7f3d2f39c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:34.474 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:34.472+0000 7f3d2f39c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:34.477 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:34.518 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:34.519 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:35.150 INFO:tasks.rados.rados.0.trial169.stdout:369: done (1 left) 2026-02-25T08:43:35.150 INFO:tasks.rados.rados.0.trial169.stdout:370: done (0 left) 2026-02-25T08:43:35.150 INFO:tasks.rados.rados.0.trial169.stdout:371: write oid 5 current snap is 53 2026-02-25T08:43:35.150 INFO:tasks.rados.rados.0.trial169.stdout:371: seq_num 188 ranges {759716=539284,2034850=614062} 2026-02-25T08:43:35.154 INFO:tasks.rados.rados.0.trial169.stdout:371: writing trial16941936-5 from 759716 to 1299000 tid 1 2026-02-25T08:43:35.160 INFO:tasks.rados.rados.0.trial169.stdout:371: writing trial16941936-5 from 2034850 to 2648912 tid 2 2026-02-25T08:43:35.160 INFO:tasks.rados.rados.0.trial169.stdout:372: read oid 28 snap -1 2026-02-25T08:43:35.160 INFO:tasks.rados.rados.0.trial169.stdout:372: expect (ObjNum 178 snap 46 seq_num 178) 2026-02-25T08:43:35.160 INFO:tasks.rados.rados.0.trial169.stdout:373: write (excl) oid 50 current snap is 53 2026-02-25T08:43:35.160 INFO:tasks.rados.rados.0.trial169.stdout:373: seq_num 189 ranges {776523=614533,1976979=757720} 2026-02-25T08:43:35.165 INFO:tasks.rados.rados.0.trial169.stdout:373: writing trial16941936-50 from 776523 to 1391056 tid 1 2026-02-25T08:43:35.171 INFO:tasks.rados.rados.0.trial169.stdout:373: writing trial16941936-50 from 1976979 to 2734699 tid 2 2026-02-25T08:43:35.171 INFO:tasks.rados.rados.0.trial169.stdout:374: read oid 31 snap -1 2026-02-25T08:43:35.171 INFO:tasks.rados.rados.0.trial169.stdout:374: expect deleted 2026-02-25T08:43:35.171 INFO:tasks.rados.rados.0.trial169.stdout:375: snap_remove snap 40 2026-02-25T08:43:36.155 INFO:tasks.rados.rados.0.trial169.stdout:375: done (4 left) 2026-02-25T08:43:36.155 INFO:tasks.rados.rados.0.trial169.stdout:376: snap_remove snap 39 2026-02-25T08:43:36.520 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:36.597 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:36.595+0000 7f8177bdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:36.600 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:36.598+0000 7f8177bdc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:36.603 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:36.644 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:36.645 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:37.159 INFO:tasks.rados.rados.0.trial169.stdout:376: done (4 left) 2026-02-25T08:43:37.160 INFO:tasks.rados.rados.0.trial169.stdout:377: write oid 10 current snap is 53 2026-02-25T08:43:37.160 INFO:tasks.rados.rados.0.trial169.stdout:377: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2026-02-25T08:43:37.165 INFO:tasks.rados.rados.0.trial169.stdout:377: writing trial16941936-10 from 793330 to 1483112 tid 1 2026-02-25T08:43:37.169 INFO:tasks.rados.rados.0.trial169.stdout:377: writing trial16941936-10 from 2035461 to 2653192 tid 2 2026-02-25T08:43:37.172 INFO:tasks.rados.rados.0.trial169.stdout:377: writing trial16941936-10 from 3101545 to 3193330 tid 3 2026-02-25T08:43:37.172 INFO:tasks.rados.rados.0.trial169.stdout:378: read oid 44 snap -1 2026-02-25T08:43:37.172 INFO:tasks.rados.rados.0.trial169.stdout:378: expect (ObjNum 186 snap 49 seq_num 186) 2026-02-25T08:43:37.172 INFO:tasks.rados.rados.0.trial169.stdout:379: write (excl) oid 15 current snap is 53 2026-02-25T08:43:37.172 INFO:tasks.rados.rados.0.trial169.stdout:371: finishing write tid 1 to trial16941936-5 2026-02-25T08:43:37.172 INFO:tasks.rados.rados.0.trial169.stdout:371: finishing write tid 2 to trial16941936-5 2026-02-25T08:43:37.172 INFO:tasks.rados.rados.0.trial169.stdout:372: expect (ObjNum 178 snap 46 seq_num 178) 2026-02-25T08:43:37.281 INFO:tasks.rados.rados.0.trial169.stdout:371: finishing write tid 3 to trial16941936-5 2026-02-25T08:43:37.281 INFO:tasks.rados.rados.0.trial169.stdout:371: finishing write tid 4 to trial16941936-5 2026-02-25T08:43:37.281 INFO:tasks.rados.rados.0.trial169.stdout:371: oid 5 updating version 0 to 40 2026-02-25T08:43:37.281 INFO:tasks.rados.rados.0.trial169.stdout:371: oid 5 version 40 is already newer than 40 2026-02-25T08:43:37.281 INFO:tasks.rados.rados.0.trial169.stdout:371: oid 5 version 40 is already newer than 40 2026-02-25T08:43:37.281 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 5 v 40 (ObjNum 188 snap 53 seq_num 188) dirty exists 2026-02-25T08:43:37.281 INFO:tasks.rados.rados.0.trial169.stdout:371: left oid 5 (ObjNum 188 snap 53 seq_num 188) 2026-02-25T08:43:37.282 INFO:tasks.rados.rados.0.trial169.stdout:373: finishing write tid 1 to trial16941936-50 2026-02-25T08:43:37.282 INFO:tasks.rados.rados.0.trial169.stdout:373: finishing write tid 2 to trial16941936-50 2026-02-25T08:43:37.282 INFO:tasks.rados.rados.0.trial169.stdout:373: finishing write tid 3 to trial16941936-50 2026-02-25T08:43:37.282 INFO:tasks.rados.rados.0.trial169.stdout:379: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2026-02-25T08:43:37.284 INFO:tasks.rados.rados.0.trial169.stdout:379: writing trial16941936-15 from 410137 to 891521 tid 1 2026-02-25T08:43:37.289 INFO:tasks.rados.rados.0.trial169.stdout:379: writing trial16941936-15 from 1410296 to 2171685 tid 2 2026-02-25T08:43:37.293 INFO:tasks.rados.rados.0.trial169.stdout:379: writing trial16941936-15 from 2728968 to 3210137 tid 3 2026-02-25T08:43:37.294 INFO:tasks.rados.rados.0.trial169.stdout:371: done (6 left) 2026-02-25T08:43:37.294 INFO:tasks.rados.rados.0.trial169.stdout:372: done (5 left) 2026-02-25T08:43:37.294 INFO:tasks.rados.rados.0.trial169.stdout:374: done (4 left) 2026-02-25T08:43:37.294 INFO:tasks.rados.rados.0.trial169.stdout:380: delete oid 22 current snap is 53 2026-02-25T08:43:37.294 INFO:tasks.rados.rados.0.trial169.stdout:373: finishing write tid 4 to trial16941936-50 2026-02-25T08:43:37.294 INFO:tasks.rados.rados.0.trial169.stdout:373: oid 50 updating version 0 to 187 2026-02-25T08:43:37.294 INFO:tasks.rados.rados.0.trial169.stdout:373: oid 50 version 187 is already newer than 185 2026-02-25T08:43:37.294 INFO:tasks.rados.rados.0.trial169.stdout:373: oid 50 version 187 is already newer than 187 2026-02-25T08:43:37.294 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 50 v 187 (ObjNum 189 snap 53 seq_num 189) dirty exists 2026-02-25T08:43:37.294 INFO:tasks.rados.rados.0.trial169.stdout:373: left oid 50 (ObjNum 189 snap 53 seq_num 189) 2026-02-25T08:43:37.295 INFO:tasks.rados.rados.0.trial169.stdout:378: expect (ObjNum 186 snap 49 seq_num 186) 2026-02-25T08:43:37.319 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 583001 2026-02-25T08:43:37.342 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:37.370 INFO:tasks.rados.rados.0.trial169.stdout:377: finishing write tid 1 to trial16941936-10 2026-02-25T08:43:37.371 INFO:tasks.rados.rados.0.trial169.stdout:377: finishing write tid 2 to trial16941936-10 2026-02-25T08:43:37.371 INFO:tasks.rados.rados.0.trial169.stdout:377: finishing write tid 3 to trial16941936-10 2026-02-25T08:43:37.371 INFO:tasks.rados.rados.0.trial169.stdout:377: finishing write tid 4 to trial16941936-10 2026-02-25T08:43:37.371 INFO:tasks.rados.rados.0.trial169.stdout:377: finishing write tid 5 to trial16941936-10 2026-02-25T08:43:37.371 INFO:tasks.rados.rados.0.trial169.stdout:377: oid 10 updating version 0 to 22 2026-02-25T08:43:37.371 INFO:tasks.rados.rados.0.trial169.stdout:377: oid 10 version 22 is already newer than 22 2026-02-25T08:43:37.371 INFO:tasks.rados.rados.0.trial169.stdout:377: oid 10 version 22 is already newer than 22 2026-02-25T08:43:37.371 INFO:tasks.rados.rados.0.trial169.stdout:377: oid 10 version 22 is already newer than 22 2026-02-25T08:43:37.371 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 10 v 22 (ObjNum 190 snap 53 seq_num 190) dirty exists 2026-02-25T08:43:37.371 INFO:tasks.rados.rados.0.trial169.stdout:377: left oid 10 (ObjNum 190 snap 53 seq_num 190) 2026-02-25T08:43:37.371 INFO:tasks.rados.rados.0.trial169.stdout:373: done (4 left) 2026-02-25T08:43:37.372 INFO:tasks.rados.rados.0.trial169.stdout:377: done (3 left) 2026-02-25T08:43:37.372 INFO:tasks.rados.rados.0.trial169.stdout:378: done (2 left) 2026-02-25T08:43:37.372 INFO:tasks.rados.rados.0.trial169.stdout:380: done (1 left) 2026-02-25T08:43:37.372 INFO:tasks.rados.rados.0.trial169.stdout:381: read oid 37 snap 46 2026-02-25T08:43:37.372 INFO:tasks.rados.rados.0.trial169.stdout:381: expect (ObjNum 129 snap 23 seq_num 129) 2026-02-25T08:43:37.372 INFO:tasks.rados.rados.0.trial169.stdout:382: read oid 49 snap 52 2026-02-25T08:43:37.372 INFO:tasks.rados.rados.0.trial169.stdout:382: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:43:37.372 INFO:tasks.rados.rados.0.trial169.stdout:383: snap_create 2026-02-25T08:43:37.372 INFO:tasks.rados.rados.0.trial169.stdout:379: finishing write tid 1 to trial16941936-15 2026-02-25T08:43:37.372 INFO:tasks.rados.rados.0.trial169.stdout:379: finishing write tid 2 to trial16941936-15 2026-02-25T08:43:37.372 INFO:tasks.rados.rados.0.trial169.stdout:379: finishing write tid 3 to trial16941936-15 2026-02-25T08:43:37.372 INFO:tasks.rados.rados.0.trial169.stdout:379: finishing write tid 4 to trial16941936-15 2026-02-25T08:43:37.373 INFO:tasks.rados.rados.0.trial169.stdout:379: finishing write tid 5 to trial16941936-15 2026-02-25T08:43:37.373 INFO:tasks.rados.rados.0.trial169.stdout:379: oid 15 updating version 0 to 93 2026-02-25T08:43:37.373 INFO:tasks.rados.rados.0.trial169.stdout:379: oid 15 version 93 is already newer than 93 2026-02-25T08:43:37.373 INFO:tasks.rados.rados.0.trial169.stdout:379: oid 15 version 93 is already newer than 90 2026-02-25T08:43:37.373 INFO:tasks.rados.rados.0.trial169.stdout:379: oid 15 version 93 is already newer than 93 2026-02-25T08:43:37.373 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 15 v 93 (ObjNum 191 snap 53 seq_num 191) dirty exists 2026-02-25T08:43:37.373 INFO:tasks.rados.rados.0.trial169.stdout:379: left oid 15 (ObjNum 191 snap 53 seq_num 191) 2026-02-25T08:43:37.384 INFO:tasks.rados.rados.0.trial169.stdout:381: expect (ObjNum 129 snap 23 seq_num 129) 2026-02-25T08:43:37.384 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:37.417 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:37.449 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 17186 2026-02-25T08:43:37.481 INFO:tasks.rados.rados.0.trial169.stdout:382: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:43:37.489 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2228609 2026-02-25T08:43:37.501 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1838106 2026-02-25T08:43:37.509 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:38.130 INFO:tasks.rados.rados.0.trial169.stdout:379: done (3 left) 2026-02-25T08:43:38.130 INFO:tasks.rados.rados.0.trial169.stdout:381: done (2 left) 2026-02-25T08:43:38.130 INFO:tasks.rados.rados.0.trial169.stdout:382: done (1 left) 2026-02-25T08:43:38.130 INFO:tasks.rados.rados.0.trial169.stdout:383: done (0 left) 2026-02-25T08:43:38.130 INFO:tasks.rados.rados.0.trial169.stdout:384: write (excl) oid 33 current snap is 54 2026-02-25T08:43:38.130 INFO:tasks.rados.rados.0.trial169.stdout:384: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2026-02-25T08:43:38.134 INFO:tasks.rados.rados.0.trial169.stdout:384: writing trial16941936-33 from 443751 to 1075633 tid 1 2026-02-25T08:43:38.140 INFO:tasks.rados.rados.0.trial169.stdout:384: writing trial16941936-33 from 1810907 to 2575965 tid 2 2026-02-25T08:43:38.142 INFO:tasks.rados.rados.0.trial169.stdout:384: writing trial16941936-33 from 3067461 to 3243751 tid 3 2026-02-25T08:43:38.142 INFO:tasks.rados.rados.0.trial169.stdout:385: write oid 9 current snap is 54 2026-02-25T08:43:38.143 INFO:tasks.rados.rados.0.trial169.stdout:385: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2026-02-25T08:43:38.147 INFO:tasks.rados.rados.0.trial169.stdout:385: writing trial16941936-9 from 460558 to 1167689 tid 1 2026-02-25T08:43:38.151 INFO:tasks.rados.rados.0.trial169.stdout:385: writing trial16941936-9 from 1869389 to 2378105 tid 2 2026-02-25T08:43:38.155 INFO:tasks.rados.rados.0.trial169.stdout:385: writing trial16941936-9 from 3094884 to 3260558 tid 3 2026-02-25T08:43:38.155 INFO:tasks.rados.rados.0.trial169.stdout:386: write (excl) oid 10 current snap is 54 2026-02-25T08:43:38.155 INFO:tasks.rados.rados.0.trial169.stdout:386: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2026-02-25T08:43:38.159 INFO:tasks.rados.rados.0.trial169.stdout:386: writing trial16941936-10 from 477365 to 1259745 tid 1 2026-02-25T08:43:38.165 INFO:tasks.rados.rados.0.trial169.stdout:386: writing trial16941936-10 from 1927871 to 2696598 tid 2 2026-02-25T08:43:38.167 INFO:tasks.rados.rados.0.trial169.stdout:386: writing trial16941936-10 from 3122307 to 3277365 tid 3 2026-02-25T08:43:38.167 INFO:tasks.rados.rados.0.trial169.stdout:387: write (excl) oid 18 current snap is 54 2026-02-25T08:43:38.168 INFO:tasks.rados.rados.0.trial169.stdout:387: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2026-02-25T08:43:38.171 INFO:tasks.rados.rados.0.trial169.stdout:387: writing trial16941936-18 from 494172 to 951801 tid 1 2026-02-25T08:43:38.175 INFO:tasks.rados.rados.0.trial169.stdout:387: writing trial16941936-18 from 1586353 to 2098738 tid 2 2026-02-25T08:43:38.180 INFO:tasks.rados.rados.0.trial169.stdout:387: writing trial16941936-18 from 2749730 to 3294172 tid 3 2026-02-25T08:43:38.180 INFO:tasks.rados.rados.0.trial169.stdout:388: snap_remove snap 49 2026-02-25T08:43:38.646 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:38.724 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:38.721+0000 7f809f10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:38.726 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:38.725+0000 7f809f10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:38.730 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:38.770 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:38.771 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:38.784 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:43:38.853 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:38.855 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:38.856 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:43:39.166 INFO:tasks.rados.rados.0.trial169.stdout:388: done (4 left) 2026-02-25T08:43:39.166 INFO:tasks.rados.rados.0.trial169.stdout:389: read oid 30 snap 52 2026-02-25T08:43:39.167 INFO:tasks.rados.rados.0.trial169.stdout:389: expect (ObjNum 147 snap 32 seq_num 147) 2026-02-25T08:43:39.167 INFO:tasks.rados.rados.0.trial169.stdout:390: read oid 22 snap 47 2026-02-25T08:43:39.167 INFO:tasks.rados.rados.0.trial169.stdout:390: expect (ObjNum 156 snap 35 seq_num 156) 2026-02-25T08:43:39.167 INFO:tasks.rados.rados.0.trial169.stdout:391: snap_create 2026-02-25T08:43:39.167 INFO:tasks.rados.rados.0.trial169.stdout:385: finishing write tid 1 to trial16941936-9 2026-02-25T08:43:39.167 INFO:tasks.rados.rados.0.trial169.stdout:385: finishing write tid 2 to trial16941936-9 2026-02-25T08:43:39.167 INFO:tasks.rados.rados.0.trial169.stdout:386: finishing write tid 1 to trial16941936-10 2026-02-25T08:43:39.167 INFO:tasks.rados.rados.0.trial169.stdout:385: finishing write tid 3 to trial16941936-9 2026-02-25T08:43:39.167 INFO:tasks.rados.rados.0.trial169.stdout:385: finishing write tid 4 to trial16941936-9 2026-02-25T08:43:39.167 INFO:tasks.rados.rados.0.trial169.stdout:385: finishing write tid 5 to trial16941936-9 2026-02-25T08:43:39.167 INFO:tasks.rados.rados.0.trial169.stdout:385: oid 9 updating version 0 to 50 2026-02-25T08:43:39.168 INFO:tasks.rados.rados.0.trial169.stdout:385: oid 9 version 50 is already newer than 50 2026-02-25T08:43:39.168 INFO:tasks.rados.rados.0.trial169.stdout:385: oid 9 version 50 is already newer than 50 2026-02-25T08:43:39.168 INFO:tasks.rados.rados.0.trial169.stdout:385: oid 9 version 50 is already newer than 50 2026-02-25T08:43:39.168 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 9 v 50 (ObjNum 194 snap 54 seq_num 194) dirty exists 2026-02-25T08:43:39.168 INFO:tasks.rados.rados.0.trial169.stdout:385: left oid 9 (ObjNum 194 snap 54 seq_num 194) 2026-02-25T08:43:39.168 INFO:tasks.rados.rados.0.trial169.stdout:387: finishing write tid 1 to trial16941936-18 2026-02-25T08:43:39.168 INFO:tasks.rados.rados.0.trial169.stdout:384: finishing write tid 1 to trial16941936-33 2026-02-25T08:43:39.168 INFO:tasks.rados.rados.0.trial169.stdout:386: finishing write tid 2 to trial16941936-10 2026-02-25T08:43:39.168 INFO:tasks.rados.rados.0.trial169.stdout:387: finishing write tid 2 to trial16941936-18 2026-02-25T08:43:39.168 INFO:tasks.rados.rados.0.trial169.stdout:386: finishing write tid 3 to trial16941936-10 2026-02-25T08:43:39.168 INFO:tasks.rados.rados.0.trial169.stdout:386: finishing write tid 4 to trial16941936-10 2026-02-25T08:43:39.168 INFO:tasks.rados.rados.0.trial169.stdout:386: finishing write tid 5 to trial16941936-10 2026-02-25T08:43:39.169 INFO:tasks.rados.rados.0.trial169.stdout:386: oid 10 updating version 0 to 27 2026-02-25T08:43:39.169 INFO:tasks.rados.rados.0.trial169.stdout:386: oid 10 version 27 is already newer than 27 2026-02-25T08:43:39.169 INFO:tasks.rados.rados.0.trial169.stdout:386: oid 10 version 27 is already newer than 24 2026-02-25T08:43:39.169 INFO:tasks.rados.rados.0.trial169.stdout:386: oid 10 version 27 is already newer than 27 2026-02-25T08:43:39.169 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 10 v 27 (ObjNum 195 snap 54 seq_num 195) dirty exists 2026-02-25T08:43:39.169 INFO:tasks.rados.rados.0.trial169.stdout:386: left oid 10 (ObjNum 195 snap 54 seq_num 195) 2026-02-25T08:43:39.169 INFO:tasks.rados.rados.0.trial169.stdout:387: finishing write tid 3 to trial16941936-18 2026-02-25T08:43:39.169 INFO:tasks.rados.rados.0.trial169.stdout:387: finishing write tid 4 to trial16941936-18 2026-02-25T08:43:39.169 INFO:tasks.rados.rados.0.trial169.stdout:387: finishing write tid 5 to trial16941936-18 2026-02-25T08:43:39.169 INFO:tasks.rados.rados.0.trial169.stdout:387: oid 18 updating version 0 to 45 2026-02-25T08:43:39.169 INFO:tasks.rados.rados.0.trial169.stdout:387: oid 18 version 45 is already newer than 42 2026-02-25T08:43:39.169 INFO:tasks.rados.rados.0.trial169.stdout:387: oid 18 version 45 is already newer than 45 2026-02-25T08:43:39.169 INFO:tasks.rados.rados.0.trial169.stdout:387: oid 18 version 45 is already newer than 45 2026-02-25T08:43:39.170 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 18 v 45 (ObjNum 196 snap 54 seq_num 196) dirty exists 2026-02-25T08:43:39.170 INFO:tasks.rados.rados.0.trial169.stdout:387: left oid 18 (ObjNum 196 snap 54 seq_num 196) 2026-02-25T08:43:39.170 INFO:tasks.rados.rados.0.trial169.stdout:384: finishing write tid 2 to trial16941936-33 2026-02-25T08:43:39.170 INFO:tasks.rados.rados.0.trial169.stdout:384: finishing write tid 3 to trial16941936-33 2026-02-25T08:43:39.170 INFO:tasks.rados.rados.0.trial169.stdout:384: finishing write tid 4 to trial16941936-33 2026-02-25T08:43:39.170 INFO:tasks.rados.rados.0.trial169.stdout:384: finishing write tid 5 to trial16941936-33 2026-02-25T08:43:39.170 INFO:tasks.rados.rados.0.trial169.stdout:384: oid 33 updating version 0 to 196 2026-02-25T08:43:39.170 INFO:tasks.rados.rados.0.trial169.stdout:384: oid 33 version 196 is already newer than 196 2026-02-25T08:43:39.170 INFO:tasks.rados.rados.0.trial169.stdout:384: oid 33 version 196 is already newer than 196 2026-02-25T08:43:39.170 INFO:tasks.rados.rados.0.trial169.stdout:384: oid 33 version 196 is already newer than 193 2026-02-25T08:43:39.170 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 33 v 196 (ObjNum 193 snap 54 seq_num 193) dirty exists 2026-02-25T08:43:39.170 INFO:tasks.rados.rados.0.trial169.stdout:384: left oid 33 (ObjNum 193 snap 54 seq_num 193) 2026-02-25T08:43:39.201 INFO:tasks.rados.rados.0.trial169.stdout:389: expect (ObjNum 147 snap 32 seq_num 147) 2026-02-25T08:43:39.272 INFO:tasks.rados.rados.0.trial169.stdout:390: expect (ObjNum 156 snap 35 seq_num 156) 2026-02-25T08:43:39.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~94s 2026-02-25T08:43:40.170 INFO:tasks.rados.rados.0.trial169.stdout:384: done (6 left) 2026-02-25T08:43:40.170 INFO:tasks.rados.rados.0.trial169.stdout:385: done (5 left) 2026-02-25T08:43:40.170 INFO:tasks.rados.rados.0.trial169.stdout:386: done (4 left) 2026-02-25T08:43:40.170 INFO:tasks.rados.rados.0.trial169.stdout:387: done (3 left) 2026-02-25T08:43:40.171 INFO:tasks.rados.rados.0.trial169.stdout:389: done (2 left) 2026-02-25T08:43:40.171 INFO:tasks.rados.rados.0.trial169.stdout:390: done (1 left) 2026-02-25T08:43:40.171 INFO:tasks.rados.rados.0.trial169.stdout:391: done (0 left) 2026-02-25T08:43:40.171 INFO:tasks.rados.rados.0.trial169.stdout:392: read oid 6 snap 47 2026-02-25T08:43:40.171 INFO:tasks.rados.rados.0.trial169.stdout:392: expect (ObjNum 57 snap 1 seq_num 57) 2026-02-25T08:43:40.171 INFO:tasks.rados.rados.0.trial169.stdout:393: snap_create 2026-02-25T08:43:40.196 INFO:tasks.rados.rados.0.trial169.stdout:392: expect (ObjNum 57 snap 1 seq_num 57) 2026-02-25T08:43:40.196 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:40.771 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:40.849 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:40.846+0000 7f94c3c3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:40.851 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:40.850+0000 7f94c3c3d640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:40.855 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:40.894 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:40.895 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:41.173 INFO:tasks.rados.rados.0.trial169.stdout:392: done (1 left) 2026-02-25T08:43:41.173 INFO:tasks.rados.rados.0.trial169.stdout:393: done (0 left) 2026-02-25T08:43:41.173 INFO:tasks.rados.rados.0.trial169.stdout:394: delete oid 23 current snap is 56 2026-02-25T08:43:41.216 INFO:tasks.rados.rados.0.trial169.stdout:394: done (0 left) 2026-02-25T08:43:41.216 INFO:tasks.rados.rados.0.trial169.stdout:395: read oid 31 snap -1 2026-02-25T08:43:41.216 INFO:tasks.rados.rados.0.trial169.stdout:395: expect deleted 2026-02-25T08:43:41.216 INFO:tasks.rados.rados.0.trial169.stdout:396: delete oid 12 current snap is 56 2026-02-25T08:43:41.219 INFO:tasks.rados.rados.0.trial169.stdout:396: done (1 left) 2026-02-25T08:43:41.219 INFO:tasks.rados.rados.0.trial169.stdout:397: write (excl) oid 37 current snap is 56 2026-02-25T08:43:41.219 INFO:tasks.rados.rados.0.trial169.stdout:397: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2026-02-25T08:43:41.224 INFO:tasks.rados.rados.0.trial169.stdout:397: writing trial16941936-37 from 544593 to 1344322 tid 1 2026-02-25T08:43:41.228 INFO:tasks.rados.rados.0.trial169.stdout:397: writing trial16941936-37 from 1761799 to 2537864 tid 2 2026-02-25T08:43:41.232 INFO:tasks.rados.rados.0.trial169.stdout:397: writing trial16941936-37 from 2948352 to 3344593 tid 3 2026-02-25T08:43:41.232 INFO:tasks.rados.rados.0.trial169.stdout:398: write oid 36 current snap is 56 2026-02-25T08:43:41.232 INFO:tasks.rados.rados.0.trial169.stdout:398: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2026-02-25T08:43:41.235 INFO:tasks.rados.rados.0.trial169.stdout:398: writing trial16941936-36 from 561400 to 1036378 tid 1 2026-02-25T08:43:41.240 INFO:tasks.rados.rados.0.trial169.stdout:398: writing trial16941936-36 from 1820281 to 2340004 tid 2 2026-02-25T08:43:41.244 INFO:tasks.rados.rados.0.trial169.stdout:398: writing trial16941936-36 from 2859422 to 3317001 tid 3 2026-02-25T08:43:41.244 INFO:tasks.rados.rados.0.trial169.stdout:399: write oid 24 current snap is 56 2026-02-25T08:43:41.244 INFO:tasks.rados.rados.0.trial169.stdout:399: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2026-02-25T08:43:41.248 INFO:tasks.rados.rados.0.trial169.stdout:399: writing trial16941936-24 from 578207 to 1128434 tid 1 2026-02-25T08:43:41.254 INFO:tasks.rados.rados.0.trial169.stdout:399: writing trial16941936-24 from 1762410 to 2542144 tid 2 2026-02-25T08:43:41.257 INFO:tasks.rados.rados.0.trial169.stdout:399: writing trial16941936-24 from 3286845 to 3378207 tid 3 2026-02-25T08:43:41.257 INFO:tasks.rados.rados.0.trial169.stdout:400: delete oid 20 current snap is 56 2026-02-25T08:43:41.260 INFO:tasks.rados.rados.0.trial169.stdout:395: done (4 left) 2026-02-25T08:43:41.260 INFO:tasks.rados.rados.0.trial169.stdout:400: done (3 left) 2026-02-25T08:43:41.260 INFO:tasks.rados.rados.0.trial169.stdout:401: write oid 7 current snap is 56 2026-02-25T08:43:41.261 INFO:tasks.rados.rados.0.trial169.stdout:401: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2026-02-25T08:43:41.265 INFO:tasks.rados.rados.0.trial169.stdout:401: writing trial16941936-7 from 611821 to 1312546 tid 1 2026-02-25T08:43:41.271 INFO:tasks.rados.rados.0.trial169.stdout:401: writing trial16941936-7 from 1879374 to 2662777 tid 2 2026-02-25T08:43:41.274 INFO:tasks.rados.rados.0.trial169.stdout:401: writing trial16941936-7 from 3341691 to 3411821 tid 3 2026-02-25T08:43:41.274 INFO:tasks.rados.rados.0.trial169.stdout:402: read oid 47 snap 44 2026-02-25T08:43:41.274 INFO:tasks.rados.rados.0.trial169.stdout:402: expect (ObjNum 46 snap 0 seq_num 46) 2026-02-25T08:43:41.274 INFO:tasks.rados.rados.0.trial169.stdout:403: write (excl) oid 38 current snap is 56 2026-02-25T08:43:41.274 INFO:tasks.rados.rados.0.trial169.stdout:403: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2026-02-25T08:43:41.279 INFO:tasks.rados.rados.0.trial169.stdout:403: writing trial16941936-38 from 628628 to 1404602 tid 1 2026-02-25T08:43:41.283 INFO:tasks.rados.rados.0.trial169.stdout:403: writing trial16941936-38 from 1937856 to 2464917 tid 2 2026-02-25T08:43:41.286 INFO:tasks.rados.rados.0.trial169.stdout:403: writing trial16941936-38 from 3252761 to 3428628 tid 3 2026-02-25T08:43:41.287 INFO:tasks.rados.rados.0.trial169.stdout:404: snap_remove snap 48 2026-02-25T08:43:42.178 INFO:tasks.rados.rados.0.trial169.stdout:404: done (6 left) 2026-02-25T08:43:42.178 INFO:tasks.rados.rados.0.trial169.stdout:405: delete oid 4 current snap is 56 2026-02-25T08:43:42.178 INFO:tasks.rados.rados.0.trial169.stdout:397: finishing write tid 1 to trial16941936-37 2026-02-25T08:43:42.178 INFO:tasks.rados.rados.0.trial169.stdout:398: finishing write tid 1 to trial16941936-36 2026-02-25T08:43:42.178 INFO:tasks.rados.rados.0.trial169.stdout:397: finishing write tid 2 to trial16941936-37 2026-02-25T08:43:42.179 INFO:tasks.rados.rados.0.trial169.stdout:399: finishing write tid 1 to trial16941936-24 2026-02-25T08:43:42.179 INFO:tasks.rados.rados.0.trial169.stdout:398: finishing write tid 2 to trial16941936-36 2026-02-25T08:43:42.179 INFO:tasks.rados.rados.0.trial169.stdout:397: finishing write tid 3 to trial16941936-37 2026-02-25T08:43:42.179 INFO:tasks.rados.rados.0.trial169.stdout:399: finishing write tid 2 to trial16941936-24 2026-02-25T08:43:42.179 INFO:tasks.rados.rados.0.trial169.stdout:398: finishing write tid 3 to trial16941936-36 2026-02-25T08:43:42.179 INFO:tasks.rados.rados.0.trial169.stdout:399: finishing write tid 3 to trial16941936-24 2026-02-25T08:43:42.179 INFO:tasks.rados.rados.0.trial169.stdout:399: finishing write tid 4 to trial16941936-24 2026-02-25T08:43:42.179 INFO:tasks.rados.rados.0.trial169.stdout:397: finishing write tid 4 to trial16941936-37 2026-02-25T08:43:42.179 INFO:tasks.rados.rados.0.trial169.stdout:399: finishing write tid 5 to trial16941936-24 2026-02-25T08:43:42.179 INFO:tasks.rados.rados.0.trial169.stdout:399: oid 24 updating version 0 to 51 2026-02-25T08:43:42.179 INFO:tasks.rados.rados.0.trial169.stdout:399: oid 24 version 51 is already newer than 51 2026-02-25T08:43:42.179 INFO:tasks.rados.rados.0.trial169.stdout:399: oid 24 version 51 is already newer than 51 2026-02-25T08:43:42.180 INFO:tasks.rados.rados.0.trial169.stdout:399: oid 24 version 51 is already newer than 51 2026-02-25T08:43:42.180 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 24 v 51 (ObjNum 201 snap 56 seq_num 201) dirty exists 2026-02-25T08:43:42.180 INFO:tasks.rados.rados.0.trial169.stdout:399: left oid 24 (ObjNum 201 snap 56 seq_num 201) 2026-02-25T08:43:42.180 INFO:tasks.rados.rados.0.trial169.stdout:397: finishing write tid 5 to trial16941936-37 2026-02-25T08:43:42.180 INFO:tasks.rados.rados.0.trial169.stdout:397: oid 37 updating version 0 to 84 2026-02-25T08:43:42.180 INFO:tasks.rados.rados.0.trial169.stdout:397: oid 37 version 84 is already newer than 81 2026-02-25T08:43:42.180 INFO:tasks.rados.rados.0.trial169.stdout:397: oid 37 version 84 is already newer than 84 2026-02-25T08:43:42.180 INFO:tasks.rados.rados.0.trial169.stdout:397: oid 37 version 84 is already newer than 84 2026-02-25T08:43:42.180 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 37 v 84 (ObjNum 199 snap 56 seq_num 199) dirty exists 2026-02-25T08:43:42.180 INFO:tasks.rados.rados.0.trial169.stdout:397: left oid 37 (ObjNum 199 snap 56 seq_num 199) 2026-02-25T08:43:42.180 INFO:tasks.rados.rados.0.trial169.stdout:398: finishing write tid 4 to trial16941936-36 2026-02-25T08:43:42.180 INFO:tasks.rados.rados.0.trial169.stdout:398: finishing write tid 5 to trial16941936-36 2026-02-25T08:43:42.181 INFO:tasks.rados.rados.0.trial169.stdout:398: oid 36 updating version 0 to 79 2026-02-25T08:43:42.181 INFO:tasks.rados.rados.0.trial169.stdout:398: oid 36 version 79 is already newer than 79 2026-02-25T08:43:42.181 INFO:tasks.rados.rados.0.trial169.stdout:398: oid 36 version 79 is already newer than 79 2026-02-25T08:43:42.181 INFO:tasks.rados.rados.0.trial169.stdout:398: oid 36 version 79 is already newer than 79 2026-02-25T08:43:42.181 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 36 v 79 (ObjNum 200 snap 56 seq_num 200) dirty exists 2026-02-25T08:43:42.181 INFO:tasks.rados.rados.0.trial169.stdout:398: left oid 36 (ObjNum 200 snap 56 seq_num 200) 2026-02-25T08:43:42.181 INFO:tasks.rados.rados.0.trial169.stdout:402: expect (ObjNum 46 snap 0 seq_num 46) 2026-02-25T08:43:42.181 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:42.235 INFO:tasks.rados.rados.0.trial169.stdout:397: done (6 left) 2026-02-25T08:43:42.235 INFO:tasks.rados.rados.0.trial169.stdout:398: done (5 left) 2026-02-25T08:43:42.235 INFO:tasks.rados.rados.0.trial169.stdout:399: done (4 left) 2026-02-25T08:43:42.235 INFO:tasks.rados.rados.0.trial169.stdout:402: done (3 left) 2026-02-25T08:43:42.235 INFO:tasks.rados.rados.0.trial169.stdout:405: done (2 left) 2026-02-25T08:43:42.236 INFO:tasks.rados.rados.0.trial169.stdout:406: read oid 22 snap 55 2026-02-25T08:43:42.236 INFO:tasks.rados.rados.0.trial169.stdout:406: expect deleted 2026-02-25T08:43:42.236 INFO:tasks.rados.rados.0.trial169.stdout:407: snap_create 2026-02-25T08:43:42.236 INFO:tasks.rados.rados.0.trial169.stdout:403: finishing write tid 1 to trial16941936-38 2026-02-25T08:43:42.236 INFO:tasks.rados.rados.0.trial169.stdout:403: finishing write tid 2 to trial16941936-38 2026-02-25T08:43:42.236 INFO:tasks.rados.rados.0.trial169.stdout:403: finishing write tid 3 to trial16941936-38 2026-02-25T08:43:42.236 INFO:tasks.rados.rados.0.trial169.stdout:403: finishing write tid 4 to trial16941936-38 2026-02-25T08:43:42.236 INFO:tasks.rados.rados.0.trial169.stdout:403: finishing write tid 5 to trial16941936-38 2026-02-25T08:43:42.236 INFO:tasks.rados.rados.0.trial169.stdout:403: oid 38 updating version 0 to 23 2026-02-25T08:43:42.236 INFO:tasks.rados.rados.0.trial169.stdout:403: oid 38 updating version 23 to 26 2026-02-25T08:43:42.236 INFO:tasks.rados.rados.0.trial169.stdout:403: oid 38 version 26 is already newer than 26 2026-02-25T08:43:42.236 INFO:tasks.rados.rados.0.trial169.stdout:403: oid 38 version 26 is already newer than 26 2026-02-25T08:43:42.237 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 38 v 26 (ObjNum 204 snap 56 seq_num 204) dirty exists 2026-02-25T08:43:42.237 INFO:tasks.rados.rados.0.trial169.stdout:403: left oid 38 (ObjNum 204 snap 56 seq_num 204) 2026-02-25T08:43:42.237 INFO:tasks.rados.rados.0.trial169.stdout:401: finishing write tid 1 to trial16941936-7 2026-02-25T08:43:42.237 INFO:tasks.rados.rados.0.trial169.stdout:401: finishing write tid 2 to trial16941936-7 2026-02-25T08:43:42.237 INFO:tasks.rados.rados.0.trial169.stdout:401: finishing write tid 3 to trial16941936-7 2026-02-25T08:43:42.237 INFO:tasks.rados.rados.0.trial169.stdout:401: finishing write tid 4 to trial16941936-7 2026-02-25T08:43:42.237 INFO:tasks.rados.rados.0.trial169.stdout:401: finishing write tid 5 to trial16941936-7 2026-02-25T08:43:42.237 INFO:tasks.rados.rados.0.trial169.stdout:401: oid 7 updating version 0 to 207 2026-02-25T08:43:42.237 INFO:tasks.rados.rados.0.trial169.stdout:401: oid 7 version 207 is already newer than 207 2026-02-25T08:43:42.237 INFO:tasks.rados.rados.0.trial169.stdout:401: oid 7 version 207 is already newer than 207 2026-02-25T08:43:42.237 INFO:tasks.rados.rados.0.trial169.stdout:401: oid 7 version 207 is already newer than 207 2026-02-25T08:43:42.237 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 7 v 207 (ObjNum 203 snap 56 seq_num 203) dirty exists 2026-02-25T08:43:42.237 INFO:tasks.rados.rados.0.trial169.stdout:401: left oid 7 (ObjNum 203 snap 56 seq_num 203) 2026-02-25T08:43:42.896 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:42.976 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:42.973+0000 7f10456c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:42.979 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:42.977+0000 7f10456c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:42.982 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:43.022 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:43.179 INFO:tasks.rados.rados.0.trial169.stdout:401: done (3 left) 2026-02-25T08:43:43.179 INFO:tasks.rados.rados.0.trial169.stdout:403: done (2 left) 2026-02-25T08:43:43.179 INFO:tasks.rados.rados.0.trial169.stdout:406: done (1 left) 2026-02-25T08:43:43.180 INFO:tasks.rados.rados.0.trial169.stdout:407: done (0 left) 2026-02-25T08:43:43.180 INFO:tasks.rados.rados.0.trial169.stdout:408: delete oid 21 current snap is 57 2026-02-25T08:43:43.224 INFO:tasks.rados.rados.0.trial169.stdout:408: done (0 left) 2026-02-25T08:43:43.224 INFO:tasks.rados.rados.0.trial169.stdout:409: snap_create 2026-02-25T08:43:43.857 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:43:43.931 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:43.934 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:43.934 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:43:44.184 INFO:tasks.rados.rados.0.trial169.stdout:409: done (0 left) 2026-02-25T08:43:44.184 INFO:tasks.rados.rados.0.trial169.stdout:410: snap_remove snap 53 2026-02-25T08:43:45.024 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:45.087 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~99s 2026-02-25T08:43:45.102 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:45.099+0000 7f0cebc11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:45.104 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:45.103+0000 7f0cebc11640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:45.108 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:45.154 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:45.185 INFO:tasks.rados.rados.0.trial169.stdout:410: done (0 left) 2026-02-25T08:43:45.185 INFO:tasks.rados.rados.0.trial169.stdout:411: read oid 22 snap -1 2026-02-25T08:43:45.185 INFO:tasks.rados.rados.0.trial169.stdout:411: expect deleted 2026-02-25T08:43:45.185 INFO:tasks.rados.rados.0.trial169.stdout:412: read oid 11 snap 50 2026-02-25T08:43:45.185 INFO:tasks.rados.rados.0.trial169.stdout:412: expect (ObjNum 187 snap 49 seq_num 187) 2026-02-25T08:43:45.185 INFO:tasks.rados.rados.0.trial169.stdout:413: write oid 3 current snap is 58 2026-02-25T08:43:45.186 INFO:tasks.rados.rados.0.trial169.stdout:413: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2026-02-25T08:43:45.190 INFO:tasks.rados.rados.0.trial169.stdout:413: writing trial16941936-3 from 679049 to 1397123 tid 1 2026-02-25T08:43:45.196 INFO:tasks.rados.rados.0.trial169.stdout:413: writing trial16941936-3 from 2113302 to 2787690 tid 2 2026-02-25T08:43:45.199 INFO:tasks.rados.rados.0.trial169.stdout:413: writing trial16941936-3 from 3335030 to 3479049 tid 3 2026-02-25T08:43:45.199 INFO:tasks.rados.rados.0.trial169.stdout:414: snap_remove snap 38 2026-02-25T08:43:46.191 INFO:tasks.rados.rados.0.trial169.stdout:414: done (3 left) 2026-02-25T08:43:46.191 INFO:tasks.rados.rados.0.trial169.stdout:415: snap_create 2026-02-25T08:43:46.191 INFO:tasks.rados.rados.0.trial169.stdout:413: finishing write tid 1 to trial16941936-3 2026-02-25T08:43:46.191 INFO:tasks.rados.rados.0.trial169.stdout:413: finishing write tid 2 to trial16941936-3 2026-02-25T08:43:46.191 INFO:tasks.rados.rados.0.trial169.stdout:413: finishing write tid 3 to trial16941936-3 2026-02-25T08:43:46.191 INFO:tasks.rados.rados.0.trial169.stdout:413: finishing write tid 4 to trial16941936-3 2026-02-25T08:43:46.191 INFO:tasks.rados.rados.0.trial169.stdout:412: expect (ObjNum 187 snap 49 seq_num 187) 2026-02-25T08:43:46.222 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:46.290 INFO:tasks.rados.rados.0.trial169.stdout:413: finishing write tid 5 to trial16941936-3 2026-02-25T08:43:46.290 INFO:tasks.rados.rados.0.trial169.stdout:413: oid 3 updating version 0 to 55 2026-02-25T08:43:46.290 INFO:tasks.rados.rados.0.trial169.stdout:413: oid 3 version 55 is already newer than 55 2026-02-25T08:43:46.290 INFO:tasks.rados.rados.0.trial169.stdout:413: oid 3 version 55 is already newer than 55 2026-02-25T08:43:46.290 INFO:tasks.rados.rados.0.trial169.stdout:413: oid 3 version 55 is already newer than 55 2026-02-25T08:43:46.290 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 3 v 55 (ObjNum 207 snap 58 seq_num 207) dirty exists 2026-02-25T08:43:46.290 INFO:tasks.rados.rados.0.trial169.stdout:413: left oid 3 (ObjNum 207 snap 58 seq_num 207) 2026-02-25T08:43:47.155 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:47.191 INFO:tasks.rados.rados.0.trial169.stdout:411: done (3 left) 2026-02-25T08:43:47.191 INFO:tasks.rados.rados.0.trial169.stdout:412: done (2 left) 2026-02-25T08:43:47.191 INFO:tasks.rados.rados.0.trial169.stdout:413: done (1 left) 2026-02-25T08:43:47.191 INFO:tasks.rados.rados.0.trial169.stdout:415: done (0 left) 2026-02-25T08:43:47.191 INFO:tasks.rados.rados.0.trial169.stdout:416: snap_create 2026-02-25T08:43:47.235 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:47.232+0000 7f21824c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:47.238 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:47.237+0000 7f21824c2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:47.242 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:47.283 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:48.132 INFO:tasks.rados.rados.0.trial169.stdout:416: done (0 left) 2026-02-25T08:43:48.133 INFO:tasks.rados.rados.0.trial169.stdout:417: snap_remove snap 52 2026-02-25T08:43:48.935 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:43:49.007 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:49.018 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:49.019 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:43:49.133 INFO:tasks.rados.rados.0.trial169.stdout:417: done (0 left) 2026-02-25T08:43:49.133 INFO:tasks.rados.rados.0.trial169.stdout:418: write oid 46 current snap is 60 2026-02-25T08:43:49.134 INFO:tasks.rados.rados.0.trial169.stdout:418: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2026-02-25T08:43:49.139 INFO:tasks.rados.rados.0.trial169.stdout:418: writing trial16941936-46 from 695856 to 1489179 tid 1 2026-02-25T08:43:49.144 INFO:tasks.rados.rados.0.trial169.stdout:418: writing trial16941936-46 from 2171784 to 2706183 tid 2 2026-02-25T08:43:49.147 INFO:tasks.rados.rados.0.trial169.stdout:418: writing trial16941936-46 from 3362453 to 3495856 tid 3 2026-02-25T08:43:49.147 INFO:tasks.rados.rados.0.trial169.stdout:419: snap_remove snap 44 2026-02-25T08:43:49.284 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:49.360 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:49.357+0000 7f9bf6ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:49.362 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:49.360+0000 7f9bf6ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:49.365 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:49.405 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:49.405 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:50.135 INFO:tasks.rados.rados.0.trial169.stdout:419: done (1 left) 2026-02-25T08:43:50.135 INFO:tasks.rados.rados.0.trial169.stdout:420: delete oid 37 current snap is 60 2026-02-25T08:43:50.135 INFO:tasks.rados.rados.0.trial169.stdout:418: finishing write tid 1 to trial16941936-46 2026-02-25T08:43:50.136 INFO:tasks.rados.rados.0.trial169.stdout:418: finishing write tid 2 to trial16941936-46 2026-02-25T08:43:50.136 INFO:tasks.rados.rados.0.trial169.stdout:418: finishing write tid 3 to trial16941936-46 2026-02-25T08:43:50.136 INFO:tasks.rados.rados.0.trial169.stdout:418: finishing write tid 4 to trial16941936-46 2026-02-25T08:43:50.136 INFO:tasks.rados.rados.0.trial169.stdout:418: finishing write tid 5 to trial16941936-46 2026-02-25T08:43:50.136 INFO:tasks.rados.rados.0.trial169.stdout:418: oid 46 updating version 0 to 88 2026-02-25T08:43:50.136 INFO:tasks.rados.rados.0.trial169.stdout:418: oid 46 version 88 is already newer than 88 2026-02-25T08:43:50.136 INFO:tasks.rados.rados.0.trial169.stdout:418: oid 46 version 88 is already newer than 88 2026-02-25T08:43:50.136 INFO:tasks.rados.rados.0.trial169.stdout:418: oid 46 version 88 is already newer than 88 2026-02-25T08:43:50.136 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 46 v 88 (ObjNum 208 snap 60 seq_num 208) dirty exists 2026-02-25T08:43:50.136 INFO:tasks.rados.rados.0.trial169.stdout:418: left oid 46 (ObjNum 208 snap 60 seq_num 208) 2026-02-25T08:43:50.174 INFO:tasks.rados.rados.0.trial169.stdout:418: done (1 left) 2026-02-25T08:43:50.175 INFO:tasks.rados.rados.0.trial169.stdout:420: done (0 left) 2026-02-25T08:43:50.175 INFO:tasks.rados.rados.0.trial169.stdout:421: delete oid 48 current snap is 60 2026-02-25T08:43:50.177 INFO:tasks.rados.rados.0.trial169.stdout:421: done (0 left) 2026-02-25T08:43:50.177 INFO:tasks.rados.rados.0.trial169.stdout:422: read oid 9 snap -1 2026-02-25T08:43:50.177 INFO:tasks.rados.rados.0.trial169.stdout:422: expect (ObjNum 194 snap 54 seq_num 194) 2026-02-25T08:43:50.178 INFO:tasks.rados.rados.0.trial169.stdout:423: read oid 33 snap 50 2026-02-25T08:43:50.178 INFO:tasks.rados.rados.0.trial169.stdout:423: expect (ObjNum 150 snap 33 seq_num 150) 2026-02-25T08:43:50.178 INFO:tasks.rados.rados.0.trial169.stdout:424: read oid 30 snap -1 2026-02-25T08:43:50.178 INFO:tasks.rados.rados.0.trial169.stdout:424: expect (ObjNum 147 snap 32 seq_num 147) 2026-02-25T08:43:50.178 INFO:tasks.rados.rados.0.trial169.stdout:425: snap_create 2026-02-25T08:43:50.190 INFO:tasks.rados.rados.0.trial169.stdout:422: expect (ObjNum 194 snap 54 seq_num 194) 2026-02-25T08:43:50.208 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:50.236 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:50.263 INFO:tasks.rados.rados.0.trial169.stdout:423: expect (ObjNum 150 snap 33 seq_num 150) 2026-02-25T08:43:50.304 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2242126 2026-02-25T08:43:50.306 INFO:tasks.rados.rados.0.trial169.stdout:424: expect (ObjNum 147 snap 32 seq_num 147) 2026-02-25T08:43:50.359 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:50.590 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~105s 2026-02-25T08:43:51.140 INFO:tasks.rados.rados.0.trial169.stdout:422: done (3 left) 2026-02-25T08:43:51.140 INFO:tasks.rados.rados.0.trial169.stdout:423: done (2 left) 2026-02-25T08:43:51.140 INFO:tasks.rados.rados.0.trial169.stdout:424: done (1 left) 2026-02-25T08:43:51.141 INFO:tasks.rados.rados.0.trial169.stdout:425: done (0 left) 2026-02-25T08:43:51.141 INFO:tasks.rados.rados.0.trial169.stdout:426: snap_remove snap 58 2026-02-25T08:43:51.406 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:51.483 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:51.480+0000 7f6e8005a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:51.486 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:51.484+0000 7f6e8005a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:51.489 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:51.529 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:51.529 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:52.143 INFO:tasks.rados.rados.0.trial169.stdout:426: done (0 left) 2026-02-25T08:43:52.143 INFO:tasks.rados.rados.0.trial169.stdout:427: write (excl) oid 30 current snap is 61 2026-02-25T08:43:52.143 INFO:tasks.rados.rados.0.trial169.stdout:427: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2026-02-25T08:43:52.149 INFO:tasks.rados.rados.0.trial169.stdout:427: writing trial16941936-30 from 746277 to 1365347 tid 1 2026-02-25T08:43:52.153 INFO:tasks.rados.rados.0.trial169.stdout:427: writing trial16941936-30 from 1830877 to 2512603 tid 2 2026-02-25T08:43:52.157 INFO:tasks.rados.rados.0.trial169.stdout:427: writing trial16941936-30 from 2928369 to 3546277 tid 3 2026-02-25T08:43:52.157 INFO:tasks.rados.rados.0.trial169.stdout:428: snap_create 2026-02-25T08:43:52.187 INFO:tasks.rados.rados.0.trial169.stdout:427: finishing write tid 1 to trial16941936-30 2026-02-25T08:43:52.202 INFO:tasks.rados.rados.0.trial169.stdout:427: finishing write tid 2 to trial16941936-30 2026-02-25T08:43:52.206 INFO:tasks.rados.rados.0.trial169.stdout:427: finishing write tid 3 to trial16941936-30 2026-02-25T08:43:52.208 INFO:tasks.rados.rados.0.trial169.stdout:427: finishing write tid 4 to trial16941936-30 2026-02-25T08:43:52.210 INFO:tasks.rados.rados.0.trial169.stdout:427: finishing write tid 5 to trial16941936-30 2026-02-25T08:43:52.211 INFO:tasks.rados.rados.0.trial169.stdout:427: oid 30 updating version 0 to 84 2026-02-25T08:43:52.211 INFO:tasks.rados.rados.0.trial169.stdout:427: oid 30 version 84 is already newer than 84 2026-02-25T08:43:52.211 INFO:tasks.rados.rados.0.trial169.stdout:427: oid 30 version 84 is already newer than 84 2026-02-25T08:43:52.211 INFO:tasks.rados.rados.0.trial169.stdout:427: oid 30 version 84 is already newer than 81 2026-02-25T08:43:52.211 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 30 v 84 (ObjNum 211 snap 61 seq_num 211) dirty exists 2026-02-25T08:43:52.211 INFO:tasks.rados.rados.0.trial169.stdout:427: left oid 30 (ObjNum 211 snap 61 seq_num 211) 2026-02-25T08:43:53.148 INFO:tasks.rados.rados.0.trial169.stdout:427: done (1 left) 2026-02-25T08:43:53.148 INFO:tasks.rados.rados.0.trial169.stdout:428: done (0 left) 2026-02-25T08:43:53.149 INFO:tasks.rados.rados.0.trial169.stdout:429: write (excl) oid 2 current snap is 62 2026-02-25T08:43:53.149 INFO:tasks.rados.rados.0.trial169.stdout:429: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2026-02-25T08:43:53.154 INFO:tasks.rados.rados.0.trial169.stdout:429: writing trial16941936-2 from 763084 to 1457403 tid 1 2026-02-25T08:43:53.157 INFO:tasks.rados.rados.0.trial169.stdout:429: writing trial16941936-2 from 1889359 to 2431096 tid 2 2026-02-25T08:43:53.162 INFO:tasks.rados.rados.0.trial169.stdout:429: writing trial16941936-2 from 2955792 to 3497694 tid 3 2026-02-25T08:43:53.162 INFO:tasks.rados.rados.0.trial169.stdout:430: read oid 35 snap -1 2026-02-25T08:43:53.162 INFO:tasks.rados.rados.0.trial169.stdout:430: expect deleted 2026-02-25T08:43:53.162 INFO:tasks.rados.rados.0.trial169.stdout:431: snap_remove snap 47 2026-02-25T08:43:53.531 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:53.610 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:53.607+0000 7f6d52ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:53.612 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:53.611+0000 7f6d52ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:53.616 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:53.657 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:54.019 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:43:54.091 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:54.095 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:43:54.151 INFO:tasks.rados.rados.0.trial169.stdout:431: done (2 left) 2026-02-25T08:43:54.151 INFO:tasks.rados.rados.0.trial169.stdout:432: write (excl) oid 9 current snap is 62 2026-02-25T08:43:54.151 INFO:tasks.rados.rados.0.trial169.stdout:432: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2026-02-25T08:43:54.155 INFO:tasks.rados.rados.0.trial169.stdout:432: writing trial16941936-9 from 779891 to 1265812 tid 1 2026-02-25T08:43:54.161 INFO:tasks.rados.rados.0.trial169.stdout:432: writing trial16941936-9 from 1947841 to 2633236 tid 2 2026-02-25T08:43:54.165 INFO:tasks.rados.rados.0.trial169.stdout:432: writing trial16941936-9 from 3383215 to 3579891 tid 3 2026-02-25T08:43:54.165 INFO:tasks.rados.rados.0.trial169.stdout:433: read oid 47 snap -1 2026-02-25T08:43:54.165 INFO:tasks.rados.rados.0.trial169.stdout:433: expect (ObjNum 46 snap 0 seq_num 46) 2026-02-25T08:43:54.165 INFO:tasks.rados.rados.0.trial169.stdout:434: snap_create 2026-02-25T08:43:54.165 INFO:tasks.rados.rados.0.trial169.stdout:429: finishing write tid 1 to trial16941936-2 2026-02-25T08:43:54.165 INFO:tasks.rados.rados.0.trial169.stdout:429: finishing write tid 2 to trial16941936-2 2026-02-25T08:43:54.166 INFO:tasks.rados.rados.0.trial169.stdout:429: finishing write tid 3 to trial16941936-2 2026-02-25T08:43:54.166 INFO:tasks.rados.rados.0.trial169.stdout:429: finishing write tid 4 to trial16941936-2 2026-02-25T08:43:54.166 INFO:tasks.rados.rados.0.trial169.stdout:429: finishing write tid 5 to trial16941936-2 2026-02-25T08:43:54.166 INFO:tasks.rados.rados.0.trial169.stdout:429: oid 2 updating version 0 to 41 2026-02-25T08:43:54.166 INFO:tasks.rados.rados.0.trial169.stdout:429: oid 2 version 41 is already newer than 41 2026-02-25T08:43:54.166 INFO:tasks.rados.rados.0.trial169.stdout:429: oid 2 version 41 is already newer than 38 2026-02-25T08:43:54.166 INFO:tasks.rados.rados.0.trial169.stdout:429: oid 2 version 41 is already newer than 41 2026-02-25T08:43:54.166 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 2 v 41 (ObjNum 212 snap 62 seq_num 212) dirty exists 2026-02-25T08:43:54.166 INFO:tasks.rados.rados.0.trial169.stdout:429: left oid 2 (ObjNum 212 snap 62 seq_num 212) 2026-02-25T08:43:54.181 INFO:tasks.rados.rados.0.trial169.stdout:433: expect (ObjNum 46 snap 0 seq_num 46) 2026-02-25T08:43:54.181 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:54.237 INFO:tasks.rados.rados.0.trial169.stdout:432: finishing write tid 1 to trial16941936-9 2026-02-25T08:43:54.237 INFO:tasks.rados.rados.0.trial169.stdout:432: finishing write tid 2 to trial16941936-9 2026-02-25T08:43:54.238 INFO:tasks.rados.rados.0.trial169.stdout:432: finishing write tid 3 to trial16941936-9 2026-02-25T08:43:54.238 INFO:tasks.rados.rados.0.trial169.stdout:432: finishing write tid 4 to trial16941936-9 2026-02-25T08:43:54.238 INFO:tasks.rados.rados.0.trial169.stdout:432: finishing write tid 5 to trial16941936-9 2026-02-25T08:43:54.238 INFO:tasks.rados.rados.0.trial169.stdout:432: oid 9 updating version 0 to 57 2026-02-25T08:43:54.238 INFO:tasks.rados.rados.0.trial169.stdout:432: oid 9 version 57 is already newer than 57 2026-02-25T08:43:54.238 INFO:tasks.rados.rados.0.trial169.stdout:432: oid 9 version 57 is already newer than 57 2026-02-25T08:43:54.238 INFO:tasks.rados.rados.0.trial169.stdout:432: oid 9 version 57 is already newer than 54 2026-02-25T08:43:54.238 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 9 v 57 (ObjNum 213 snap 62 seq_num 213) dirty exists 2026-02-25T08:43:54.238 INFO:tasks.rados.rados.0.trial169.stdout:432: left oid 9 (ObjNum 213 snap 62 seq_num 213) 2026-02-25T08:43:55.155 INFO:tasks.rados.rados.0.trial169.stdout:429: done (4 left) 2026-02-25T08:43:55.155 INFO:tasks.rados.rados.0.trial169.stdout:430: done (3 left) 2026-02-25T08:43:55.155 INFO:tasks.rados.rados.0.trial169.stdout:432: done (2 left) 2026-02-25T08:43:55.155 INFO:tasks.rados.rados.0.trial169.stdout:433: done (1 left) 2026-02-25T08:43:55.155 INFO:tasks.rados.rados.0.trial169.stdout:434: done (0 left) 2026-02-25T08:43:55.155 INFO:tasks.rados.rados.0.trial169.stdout:435: read oid 7 snap -1 2026-02-25T08:43:55.155 INFO:tasks.rados.rados.0.trial169.stdout:435: expect (ObjNum 203 snap 56 seq_num 203) 2026-02-25T08:43:55.156 INFO:tasks.rados.rados.0.trial169.stdout:436: write oid 10 current snap is 63 2026-02-25T08:43:55.156 INFO:tasks.rados.rados.0.trial169.stdout:436: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2026-02-25T08:43:55.159 INFO:tasks.rados.rados.0.trial169.stdout:436: writing trial16941936-10 from 796698 to 1357868 tid 1 2026-02-25T08:43:55.164 INFO:tasks.rados.rados.0.trial169.stdout:436: writing trial16941936-10 from 2006323 to 2551729 tid 2 2026-02-25T08:43:55.168 INFO:tasks.rados.rados.0.trial169.stdout:436: writing trial16941936-10 from 3126991 to 3596698 tid 3 2026-02-25T08:43:55.168 INFO:tasks.rados.rados.0.trial169.stdout:437: write (excl) oid 38 current snap is 63 2026-02-25T08:43:55.168 INFO:tasks.rados.rados.0.trial169.stdout:437: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2026-02-25T08:43:55.172 INFO:tasks.rados.rados.0.trial169.stdout:437: writing trial16941936-38 from 413505 to 1049924 tid 1 2026-02-25T08:43:55.177 INFO:tasks.rados.rados.0.trial169.stdout:437: writing trial16941936-38 from 1664805 to 2353869 tid 2 2026-02-25T08:43:55.182 INFO:tasks.rados.rados.0.trial169.stdout:437: writing trial16941936-38 from 3038061 to 3530132 tid 3 2026-02-25T08:43:55.182 INFO:tasks.rados.rados.0.trial169.stdout:438: write (excl) oid 26 current snap is 63 2026-02-25T08:43:55.182 INFO:tasks.rados.rados.0.trial169.stdout:438: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2026-02-25T08:43:55.186 INFO:tasks.rados.rados.0.trial169.stdout:438: writing trial16941936-26 from 430312 to 1141980 tid 1 2026-02-25T08:43:55.190 INFO:tasks.rados.rados.0.trial169.stdout:438: writing trial16941936-26 from 1723287 to 2272362 tid 2 2026-02-25T08:43:55.195 INFO:tasks.rados.rados.0.trial169.stdout:438: writing trial16941936-26 from 2781837 to 3485180 tid 3 2026-02-25T08:43:55.195 INFO:tasks.rados.rados.0.trial169.stdout:439: read oid 22 snap -1 2026-02-25T08:43:55.195 INFO:tasks.rados.rados.0.trial169.stdout:439: expect deleted 2026-02-25T08:43:55.196 INFO:tasks.rados.rados.0.trial169.stdout:440: read oid 12 snap 27 2026-02-25T08:43:55.196 INFO:tasks.rados.rados.0.trial169.stdout:440: expect (ObjNum 123 snap 17 seq_num 123) 2026-02-25T08:43:55.196 INFO:tasks.rados.rados.0.trial169.stdout:441: delete oid 35 current snap is 63 2026-02-25T08:43:55.207 INFO:tasks.rados.rados.0.trial169.stdout:437: finishing write tid 1 to trial16941936-38 2026-02-25T08:43:55.213 INFO:tasks.rados.rados.0.trial169.stdout:436: finishing write tid 1 to trial16941936-10 2026-02-25T08:43:55.225 INFO:tasks.rados.rados.0.trial169.stdout:440: expect (ObjNum 123 snap 17 seq_num 123) 2026-02-25T08:43:55.228 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 770690 2026-02-25T08:43:55.246 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 223221 2026-02-25T08:43:55.293 INFO:tasks.rados.rados.0.trial169.stdout:436: finishing write tid 2 to trial16941936-10 2026-02-25T08:43:55.293 INFO:tasks.rados.rados.0.trial169.stdout:437: finishing write tid 2 to trial16941936-38 2026-02-25T08:43:55.293 INFO:tasks.rados.rados.0.trial169.stdout:436: finishing write tid 3 to trial16941936-10 2026-02-25T08:43:55.293 INFO:tasks.rados.rados.0.trial169.stdout:437: finishing write tid 3 to trial16941936-38 2026-02-25T08:43:55.293 INFO:tasks.rados.rados.0.trial169.stdout:436: finishing write tid 4 to trial16941936-10 2026-02-25T08:43:55.293 INFO:tasks.rados.rados.0.trial169.stdout:439: done (6 left) 2026-02-25T08:43:55.293 INFO:tasks.rados.rados.0.trial169.stdout:440: done (5 left) 2026-02-25T08:43:55.293 INFO:tasks.rados.rados.0.trial169.stdout:441: done (4 left) 2026-02-25T08:43:55.293 INFO:tasks.rados.rados.0.trial169.stdout:442: write oid 9 current snap is 63 2026-02-25T08:43:55.293 INFO:tasks.rados.rados.0.trial169.stdout:436: finishing write tid 5 to trial16941936-10 2026-02-25T08:43:55.294 INFO:tasks.rados.rados.0.trial169.stdout:436: oid 10 updating version 0 to 46 2026-02-25T08:43:55.294 INFO:tasks.rados.rados.0.trial169.stdout:436: oid 10 version 46 is already newer than 46 2026-02-25T08:43:55.294 INFO:tasks.rados.rados.0.trial169.stdout:436: oid 10 version 46 is already newer than 46 2026-02-25T08:43:55.294 INFO:tasks.rados.rados.0.trial169.stdout:436: oid 10 version 46 is already newer than 46 2026-02-25T08:43:55.294 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 10 v 46 (ObjNum 214 snap 63 seq_num 214) dirty exists 2026-02-25T08:43:55.294 INFO:tasks.rados.rados.0.trial169.stdout:436: left oid 10 (ObjNum 214 snap 63 seq_num 214) 2026-02-25T08:43:55.294 INFO:tasks.rados.rados.0.trial169.stdout:437: finishing write tid 4 to trial16941936-38 2026-02-25T08:43:55.294 INFO:tasks.rados.rados.0.trial169.stdout:437: finishing write tid 5 to trial16941936-38 2026-02-25T08:43:55.294 INFO:tasks.rados.rados.0.trial169.stdout:437: oid 38 updating version 0 to 31 2026-02-25T08:43:55.294 INFO:tasks.rados.rados.0.trial169.stdout:437: oid 38 version 31 is already newer than 31 2026-02-25T08:43:55.294 INFO:tasks.rados.rados.0.trial169.stdout:437: oid 38 version 31 is already newer than 28 2026-02-25T08:43:55.294 INFO:tasks.rados.rados.0.trial169.stdout:437: oid 38 version 31 is already newer than 31 2026-02-25T08:43:55.294 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 38 v 31 (ObjNum 215 snap 63 seq_num 215) dirty exists 2026-02-25T08:43:55.295 INFO:tasks.rados.rados.0.trial169.stdout:437: left oid 38 (ObjNum 215 snap 63 seq_num 215) 2026-02-25T08:43:55.295 INFO:tasks.rados.rados.0.trial169.stdout:435: expect (ObjNum 203 snap 56 seq_num 203) 2026-02-25T08:43:55.380 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:55.424 INFO:tasks.rados.rados.0.trial169.stdout:438: finishing write tid 1 to trial16941936-26 2026-02-25T08:43:55.424 INFO:tasks.rados.rados.0.trial169.stdout:438: finishing write tid 2 to trial16941936-26 2026-02-25T08:43:55.424 INFO:tasks.rados.rados.0.trial169.stdout:438: finishing write tid 3 to trial16941936-26 2026-02-25T08:43:55.424 INFO:tasks.rados.rados.0.trial169.stdout:438: finishing write tid 4 to trial16941936-26 2026-02-25T08:43:55.424 INFO:tasks.rados.rados.0.trial169.stdout:438: finishing write tid 5 to trial16941936-26 2026-02-25T08:43:55.424 INFO:tasks.rados.rados.0.trial169.stdout:438: oid 26 updating version 0 to 166 2026-02-25T08:43:55.424 INFO:tasks.rados.rados.0.trial169.stdout:438: oid 26 version 166 is already newer than 163 2026-02-25T08:43:55.424 INFO:tasks.rados.rados.0.trial169.stdout:438: oid 26 version 166 is already newer than 166 2026-02-25T08:43:55.425 INFO:tasks.rados.rados.0.trial169.stdout:438: oid 26 version 166 is already newer than 166 2026-02-25T08:43:55.425 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 26 v 166 (ObjNum 216 snap 63 seq_num 216) dirty exists 2026-02-25T08:43:55.425 INFO:tasks.rados.rados.0.trial169.stdout:438: left oid 26 (ObjNum 216 snap 63 seq_num 216) 2026-02-25T08:43:55.425 INFO:tasks.rados.rados.0.trial169.stdout:442: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2026-02-25T08:43:55.427 INFO:tasks.rados.rados.0.trial169.stdout:442: writing trial16941936-9 from 463926 to 926092 tid 1 2026-02-25T08:43:55.432 INFO:tasks.rados.rados.0.trial169.stdout:442: writing trial16941936-9 from 1723898 to 2160289 tid 2 2026-02-25T08:43:55.437 INFO:tasks.rados.rados.0.trial169.stdout:442: writing trial16941936-9 from 2603977 to 3329864 tid 3 2026-02-25T08:43:55.437 INFO:tasks.rados.rados.0.trial169.stdout:435: done (4 left) 2026-02-25T08:43:55.437 INFO:tasks.rados.rados.0.trial169.stdout:436: done (3 left) 2026-02-25T08:43:55.437 INFO:tasks.rados.rados.0.trial169.stdout:437: done (2 left) 2026-02-25T08:43:55.437 INFO:tasks.rados.rados.0.trial169.stdout:438: done (1 left) 2026-02-25T08:43:55.437 INFO:tasks.rados.rados.0.trial169.stdout:443: snap_create 2026-02-25T08:43:55.458 INFO:tasks.rados.rados.0.trial169.stdout:442: finishing write tid 1 to trial16941936-9 2026-02-25T08:43:55.463 INFO:tasks.rados.rados.0.trial169.stdout:442: finishing write tid 2 to trial16941936-9 2026-02-25T08:43:55.468 INFO:tasks.rados.rados.0.trial169.stdout:442: finishing write tid 3 to trial16941936-9 2026-02-25T08:43:55.471 INFO:tasks.rados.rados.0.trial169.stdout:442: finishing write tid 4 to trial16941936-9 2026-02-25T08:43:55.473 INFO:tasks.rados.rados.0.trial169.stdout:442: finishing write tid 5 to trial16941936-9 2026-02-25T08:43:55.473 INFO:tasks.rados.rados.0.trial169.stdout:442: oid 9 updating version 0 to 62 2026-02-25T08:43:55.473 INFO:tasks.rados.rados.0.trial169.stdout:442: oid 9 version 62 is already newer than 62 2026-02-25T08:43:55.474 INFO:tasks.rados.rados.0.trial169.stdout:442: oid 9 version 62 is already newer than 62 2026-02-25T08:43:55.474 INFO:tasks.rados.rados.0.trial169.stdout:442: oid 9 version 62 is already newer than 62 2026-02-25T08:43:55.474 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 9 v 62 (ObjNum 218 snap 63 seq_num 218) dirty exists 2026-02-25T08:43:55.474 INFO:tasks.rados.rados.0.trial169.stdout:442: left oid 9 (ObjNum 218 snap 63 seq_num 218) 2026-02-25T08:43:55.657 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:55.734 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:55.731+0000 7f4633ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:55.736 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:55.734+0000 7f4633ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:55.740 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:55.780 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:55.780 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:56.095 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~110s 2026-02-25T08:43:56.159 INFO:tasks.rados.rados.0.trial169.stdout:442: done (1 left) 2026-02-25T08:43:56.159 INFO:tasks.rados.rados.0.trial169.stdout:443: done (0 left) 2026-02-25T08:43:56.159 INFO:tasks.rados.rados.0.trial169.stdout:444: write oid 8 current snap is 64 2026-02-25T08:43:56.159 INFO:tasks.rados.rados.0.trial169.stdout:444: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2026-02-25T08:43:56.162 INFO:tasks.rados.rados.0.trial169.stdout:444: writing trial16941936-8 from 480733 to 1018148 tid 1 2026-02-25T08:43:56.168 INFO:tasks.rados.rados.0.trial169.stdout:444: writing trial16941936-8 from 1782380 to 2478782 tid 2 2026-02-25T08:43:56.173 INFO:tasks.rados.rados.0.trial169.stdout:444: writing trial16941936-8 from 3031400 to 3568559 tid 3 2026-02-25T08:43:56.173 INFO:tasks.rados.rados.0.trial169.stdout:445: read oid 30 snap -1 2026-02-25T08:43:56.173 INFO:tasks.rados.rados.0.trial169.stdout:445: expect (ObjNum 211 snap 61 seq_num 211) 2026-02-25T08:43:56.173 INFO:tasks.rados.rados.0.trial169.stdout:446: read oid 34 snap 56 2026-02-25T08:43:56.173 INFO:tasks.rados.rados.0.trial169.stdout:446: expect (ObjNum 145 snap 32 seq_num 145) 2026-02-25T08:43:56.173 INFO:tasks.rados.rados.0.trial169.stdout:447: write oid 44 current snap is 64 2026-02-25T08:43:56.173 INFO:tasks.rados.rados.0.trial169.stdout:447: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2026-02-25T08:43:56.176 INFO:tasks.rados.rados.0.trial169.stdout:447: writing trial16941936-44 from 497540 to 1110204 tid 1 2026-02-25T08:43:56.181 INFO:tasks.rados.rados.0.trial169.stdout:447: writing trial16941936-44 from 1840862 to 2280922 tid 2 2026-02-25T08:43:56.186 INFO:tasks.rados.rados.0.trial169.stdout:447: writing trial16941936-44 from 3058823 to 3523607 tid 3 2026-02-25T08:43:56.186 INFO:tasks.rados.rados.0.trial169.stdout:448: read oid 45 snap -1 2026-02-25T08:43:56.186 INFO:tasks.rados.rados.0.trial169.stdout:448: expect (ObjNum 173 snap 43 seq_num 173) 2026-02-25T08:43:56.186 INFO:tasks.rados.rados.0.trial169.stdout:449: read oid 47 snap -1 2026-02-25T08:43:56.186 INFO:tasks.rados.rados.0.trial169.stdout:449: expect (ObjNum 46 snap 0 seq_num 46) 2026-02-25T08:43:56.186 INFO:tasks.rados.rados.0.trial169.stdout:450: write (excl) oid 17 current snap is 64 2026-02-25T08:43:56.186 INFO:tasks.rados.rados.0.trial169.stdout:450: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2026-02-25T08:43:56.189 INFO:tasks.rados.rados.0.trial169.stdout:450: writing trial16941936-17 from 514347 to 918613 tid 1 2026-02-25T08:43:56.194 INFO:tasks.rados.rados.0.trial169.stdout:450: writing trial16941936-17 from 1499344 to 2199415 tid 2 2026-02-25T08:43:56.198 INFO:tasks.rados.rados.0.trial169.stdout:450: writing trial16941936-17 from 2686246 to 3362302 tid 3 2026-02-25T08:43:56.199 INFO:tasks.rados.rados.0.trial169.stdout:451: snap_create 2026-02-25T08:43:56.199 INFO:tasks.rados.rados.0.trial169.stdout:446: expect (ObjNum 145 snap 32 seq_num 145) 2026-02-25T08:43:56.253 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1812790 2026-02-25T08:43:56.266 INFO:tasks.rados.rados.0.trial169.stdout:448: expect (ObjNum 173 snap 43 seq_num 173) 2026-02-25T08:43:56.270 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 805473 2026-02-25T08:43:56.305 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 128883 2026-02-25T08:43:56.375 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2785473 2026-02-25T08:43:56.379 INFO:tasks.rados.rados.0.trial169.stdout:449: expect (ObjNum 46 snap 0 seq_num 46) 2026-02-25T08:43:56.379 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:56.421 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:43:56.445 INFO:tasks.rados.rados.0.trial169.stdout:445: expect (ObjNum 211 snap 61 seq_num 211) 2026-02-25T08:43:56.565 INFO:tasks.rados.rados.0.trial169.stdout:450: finishing write tid 1 to trial16941936-17 2026-02-25T08:43:56.565 INFO:tasks.rados.rados.0.trial169.stdout:450: finishing write tid 2 to trial16941936-17 2026-02-25T08:43:56.566 INFO:tasks.rados.rados.0.trial169.stdout:450: finishing write tid 3 to trial16941936-17 2026-02-25T08:43:56.566 INFO:tasks.rados.rados.0.trial169.stdout:444: finishing write tid 1 to trial16941936-8 2026-02-25T08:43:56.566 INFO:tasks.rados.rados.0.trial169.stdout:450: finishing write tid 4 to trial16941936-17 2026-02-25T08:43:56.566 INFO:tasks.rados.rados.0.trial169.stdout:450: finishing write tid 5 to trial16941936-17 2026-02-25T08:43:56.566 INFO:tasks.rados.rados.0.trial169.stdout:450: oid 17 updating version 0 to 43 2026-02-25T08:43:56.566 INFO:tasks.rados.rados.0.trial169.stdout:450: oid 17 updating version 43 to 46 2026-02-25T08:43:56.566 INFO:tasks.rados.rados.0.trial169.stdout:450: oid 17 version 46 is already newer than 46 2026-02-25T08:43:56.566 INFO:tasks.rados.rados.0.trial169.stdout:450: oid 17 version 46 is already newer than 46 2026-02-25T08:43:56.566 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 17 v 46 (ObjNum 221 snap 64 seq_num 221) dirty exists 2026-02-25T08:43:56.566 INFO:tasks.rados.rados.0.trial169.stdout:450: left oid 17 (ObjNum 221 snap 64 seq_num 221) 2026-02-25T08:43:56.566 INFO:tasks.rados.rados.0.trial169.stdout:444: finishing write tid 2 to trial16941936-8 2026-02-25T08:43:56.567 INFO:tasks.rados.rados.0.trial169.stdout:444: finishing write tid 3 to trial16941936-8 2026-02-25T08:43:56.567 INFO:tasks.rados.rados.0.trial169.stdout:444: finishing write tid 4 to trial16941936-8 2026-02-25T08:43:56.567 INFO:tasks.rados.rados.0.trial169.stdout:444: finishing write tid 5 to trial16941936-8 2026-02-25T08:43:56.567 INFO:tasks.rados.rados.0.trial169.stdout:444: oid 8 updating version 0 to 171 2026-02-25T08:43:56.567 INFO:tasks.rados.rados.0.trial169.stdout:444: oid 8 version 171 is already newer than 171 2026-02-25T08:43:56.567 INFO:tasks.rados.rados.0.trial169.stdout:444: oid 8 version 171 is already newer than 171 2026-02-25T08:43:56.567 INFO:tasks.rados.rados.0.trial169.stdout:444: oid 8 version 171 is already newer than 171 2026-02-25T08:43:56.567 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 8 v 171 (ObjNum 219 snap 64 seq_num 219) dirty exists 2026-02-25T08:43:56.567 INFO:tasks.rados.rados.0.trial169.stdout:444: left oid 8 (ObjNum 219 snap 64 seq_num 219) 2026-02-25T08:43:56.567 INFO:tasks.rados.rados.0.trial169.stdout:447: finishing write tid 1 to trial16941936-44 2026-02-25T08:43:56.567 INFO:tasks.rados.rados.0.trial169.stdout:447: finishing write tid 2 to trial16941936-44 2026-02-25T08:43:56.567 INFO:tasks.rados.rados.0.trial169.stdout:447: finishing write tid 3 to trial16941936-44 2026-02-25T08:43:56.567 INFO:tasks.rados.rados.0.trial169.stdout:447: finishing write tid 4 to trial16941936-44 2026-02-25T08:43:56.568 INFO:tasks.rados.rados.0.trial169.stdout:447: finishing write tid 5 to trial16941936-44 2026-02-25T08:43:56.568 INFO:tasks.rados.rados.0.trial169.stdout:447: oid 44 updating version 0 to 89 2026-02-25T08:43:56.568 INFO:tasks.rados.rados.0.trial169.stdout:447: oid 44 version 89 is already newer than 89 2026-02-25T08:43:56.568 INFO:tasks.rados.rados.0.trial169.stdout:447: oid 44 version 89 is already newer than 89 2026-02-25T08:43:56.568 INFO:tasks.rados.rados.0.trial169.stdout:447: oid 44 version 89 is already newer than 89 2026-02-25T08:43:56.568 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 44 v 89 (ObjNum 220 snap 64 seq_num 220) dirty exists 2026-02-25T08:43:56.568 INFO:tasks.rados.rados.0.trial169.stdout:447: left oid 44 (ObjNum 220 snap 64 seq_num 220) 2026-02-25T08:43:57.163 INFO:tasks.rados.rados.0.trial169.stdout:444: done (7 left) 2026-02-25T08:43:57.163 INFO:tasks.rados.rados.0.trial169.stdout:445: done (6 left) 2026-02-25T08:43:57.163 INFO:tasks.rados.rados.0.trial169.stdout:446: done (5 left) 2026-02-25T08:43:57.163 INFO:tasks.rados.rados.0.trial169.stdout:447: done (4 left) 2026-02-25T08:43:57.163 INFO:tasks.rados.rados.0.trial169.stdout:448: done (3 left) 2026-02-25T08:43:57.163 INFO:tasks.rados.rados.0.trial169.stdout:449: done (2 left) 2026-02-25T08:43:57.163 INFO:tasks.rados.rados.0.trial169.stdout:450: done (1 left) 2026-02-25T08:43:57.163 INFO:tasks.rados.rados.0.trial169.stdout:451: done (0 left) 2026-02-25T08:43:57.163 INFO:tasks.rados.rados.0.trial169.stdout:452: delete oid 4 current snap is 65 2026-02-25T08:43:57.177 INFO:tasks.rados.rados.0.trial169.stdout:452: done (0 left) 2026-02-25T08:43:57.177 INFO:tasks.rados.rados.0.trial169.stdout:453: delete oid 1 current snap is 65 2026-02-25T08:43:57.205 INFO:tasks.rados.rados.0.trial169.stdout:453: done (0 left) 2026-02-25T08:43:57.206 INFO:tasks.rados.rados.0.trial169.stdout:454: read oid 41 snap 27 2026-02-25T08:43:57.206 INFO:tasks.rados.rados.0.trial169.stdout:454: expect (ObjNum 137 snap 26 seq_num 137) 2026-02-25T08:43:57.206 INFO:tasks.rados.rados.0.trial169.stdout:455: write oid 3 current snap is 65 2026-02-25T08:43:57.206 INFO:tasks.rados.rados.0.trial169.stdout:455: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2026-02-25T08:43:57.210 INFO:tasks.rados.rados.0.trial169.stdout:455: writing trial16941936-3 from 564768 to 1194781 tid 1 2026-02-25T08:43:57.214 INFO:tasks.rados.rados.0.trial169.stdout:455: writing trial16941936-3 from 1674790 to 2122188 tid 2 2026-02-25T08:43:57.218 INFO:tasks.rados.rados.0.trial169.stdout:455: writing trial16941936-3 from 2768515 to 3394740 tid 3 2026-02-25T08:43:57.219 INFO:tasks.rados.rados.0.trial169.stdout:456: write oid 47 current snap is 65 2026-02-25T08:43:57.219 INFO:tasks.rados.rados.0.trial169.stdout:456: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2026-02-25T08:43:57.223 INFO:tasks.rados.rados.0.trial169.stdout:456: writing trial16941936-47 from 581575 to 1286837 tid 1 2026-02-25T08:43:57.227 INFO:tasks.rados.rados.0.trial169.stdout:456: writing trial16941936-47 from 1733272 to 2440681 tid 2 2026-02-25T08:43:57.232 INFO:tasks.rados.rados.0.trial169.stdout:456: writing trial16941936-47 from 3195938 to 3633435 tid 3 2026-02-25T08:43:57.232 INFO:tasks.rados.rados.0.trial169.stdout:457: snap_create 2026-02-25T08:43:57.232 INFO:tasks.rados.rados.0.trial169.stdout:454: expect (ObjNum 137 snap 26 seq_num 137) 2026-02-25T08:43:57.275 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1743391 2026-02-25T08:43:57.319 INFO:tasks.rados.rados.0.trial169.stdout:455: finishing write tid 1 to trial16941936-3 2026-02-25T08:43:57.320 INFO:tasks.rados.rados.0.trial169.stdout:455: finishing write tid 2 to trial16941936-3 2026-02-25T08:43:57.320 INFO:tasks.rados.rados.0.trial169.stdout:455: finishing write tid 3 to trial16941936-3 2026-02-25T08:43:57.320 INFO:tasks.rados.rados.0.trial169.stdout:455: finishing write tid 4 to trial16941936-3 2026-02-25T08:43:57.320 INFO:tasks.rados.rados.0.trial169.stdout:456: finishing write tid 1 to trial16941936-47 2026-02-25T08:43:57.320 INFO:tasks.rados.rados.0.trial169.stdout:455: finishing write tid 5 to trial16941936-3 2026-02-25T08:43:57.320 INFO:tasks.rados.rados.0.trial169.stdout:455: oid 3 updating version 0 to 68 2026-02-25T08:43:57.320 INFO:tasks.rados.rados.0.trial169.stdout:455: oid 3 version 68 is already newer than 68 2026-02-25T08:43:57.320 INFO:tasks.rados.rados.0.trial169.stdout:455: oid 3 version 68 is already newer than 68 2026-02-25T08:43:57.320 INFO:tasks.rados.rados.0.trial169.stdout:455: oid 3 version 68 is already newer than 68 2026-02-25T08:43:57.320 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 3 v 68 (ObjNum 224 snap 65 seq_num 224) dirty exists 2026-02-25T08:43:57.320 INFO:tasks.rados.rados.0.trial169.stdout:455: left oid 3 (ObjNum 224 snap 65 seq_num 224) 2026-02-25T08:43:57.321 INFO:tasks.rados.rados.0.trial169.stdout:456: finishing write tid 2 to trial16941936-47 2026-02-25T08:43:57.321 INFO:tasks.rados.rados.0.trial169.stdout:456: finishing write tid 3 to trial16941936-47 2026-02-25T08:43:57.321 INFO:tasks.rados.rados.0.trial169.stdout:456: finishing write tid 4 to trial16941936-47 2026-02-25T08:43:57.321 INFO:tasks.rados.rados.0.trial169.stdout:456: finishing write tid 5 to trial16941936-47 2026-02-25T08:43:57.321 INFO:tasks.rados.rados.0.trial169.stdout:456: oid 47 updating version 0 to 38 2026-02-25T08:43:57.321 INFO:tasks.rados.rados.0.trial169.stdout:456: oid 47 version 38 is already newer than 38 2026-02-25T08:43:57.321 INFO:tasks.rados.rados.0.trial169.stdout:456: oid 47 version 38 is already newer than 38 2026-02-25T08:43:57.321 INFO:tasks.rados.rados.0.trial169.stdout:456: oid 47 version 38 is already newer than 38 2026-02-25T08:43:57.321 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 47 v 38 (ObjNum 225 snap 65 seq_num 225) dirty exists 2026-02-25T08:43:57.321 INFO:tasks.rados.rados.0.trial169.stdout:456: left oid 47 (ObjNum 225 snap 65 seq_num 225) 2026-02-25T08:43:57.782 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:43:57.860 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:57.857+0000 7fc7cb1dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:57.863 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:57.861+0000 7fc7cb1dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:57.866 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:43:57.907 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:43:57.907 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:43:58.134 INFO:tasks.rados.rados.0.trial169.stdout:454: done (3 left) 2026-02-25T08:43:58.134 INFO:tasks.rados.rados.0.trial169.stdout:455: done (2 left) 2026-02-25T08:43:58.134 INFO:tasks.rados.rados.0.trial169.stdout:456: done (1 left) 2026-02-25T08:43:58.134 INFO:tasks.rados.rados.0.trial169.stdout:457: done (0 left) 2026-02-25T08:43:58.134 INFO:tasks.rados.rados.0.trial169.stdout:458: delete oid 13 current snap is 66 2026-02-25T08:43:58.149 INFO:tasks.rados.rados.0.trial169.stdout:458: done (0 left) 2026-02-25T08:43:58.149 INFO:tasks.rados.rados.0.trial169.stdout:459: delete oid 48 current snap is 66 2026-02-25T08:43:58.150 INFO:tasks.rados.rados.0.trial169.stdout:459: done (0 left) 2026-02-25T08:43:58.151 INFO:tasks.rados.rados.0.trial169.stdout:460: read oid 40 snap -1 2026-02-25T08:43:58.151 INFO:tasks.rados.rados.0.trial169.stdout:460: expect (ObjNum 183 snap 46 seq_num 183) 2026-02-25T08:43:58.151 INFO:tasks.rados.rados.0.trial169.stdout:461: read oid 9 snap -1 2026-02-25T08:43:58.151 INFO:tasks.rados.rados.0.trial169.stdout:461: expect (ObjNum 218 snap 63 seq_num 218) 2026-02-25T08:43:58.151 INFO:tasks.rados.rados.0.trial169.stdout:462: write (excl) oid 24 current snap is 66 2026-02-25T08:43:58.151 INFO:tasks.rados.rados.0.trial169.stdout:462: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2026-02-25T08:43:58.155 INFO:tasks.rados.rados.0.trial169.stdout:462: writing trial16941936-24 from 631996 to 1163005 tid 1 2026-02-25T08:43:58.159 INFO:tasks.rados.rados.0.trial169.stdout:462: writing trial16941936-24 from 1792365 to 2247101 tid 2 2026-02-25T08:43:58.164 INFO:tasks.rados.rados.0.trial169.stdout:462: writing trial16941936-24 from 2761854 to 3433167 tid 3 2026-02-25T08:43:58.164 INFO:tasks.rados.rados.0.trial169.stdout:463: delete oid 16 current snap is 66 2026-02-25T08:43:58.168 INFO:tasks.rados.rados.0.trial169.stdout:463: done (3 left) 2026-02-25T08:43:58.168 INFO:tasks.rados.rados.0.trial169.stdout:464: read oid 8 snap 46 2026-02-25T08:43:58.168 INFO:tasks.rados.rados.0.trial169.stdout:464: expect (ObjNum 149 snap 32 seq_num 149) 2026-02-25T08:43:58.168 INFO:tasks.rados.rados.0.trial169.stdout:465: read oid 1 snap -1 2026-02-25T08:43:58.169 INFO:tasks.rados.rados.0.trial169.stdout:465: expect deleted 2026-02-25T08:43:58.169 INFO:tasks.rados.rados.0.trial169.stdout:466: write oid 14 current snap is 66 2026-02-25T08:43:58.169 INFO:tasks.rados.rados.0.trial169.stdout:466: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2026-02-25T08:43:58.174 INFO:tasks.rados.rados.0.trial169.stdout:466: writing trial16941936-14 from 665610 to 1463470 tid 1 2026-02-25T08:43:58.177 INFO:tasks.rados.rados.0.trial169.stdout:466: writing trial16941936-14 from 1909329 to 2367734 tid 2 2026-02-25T08:43:58.180 INFO:tasks.rados.rados.0.trial169.stdout:466: writing trial16941936-14 from 2816700 to 3226910 tid 3 2026-02-25T08:43:58.181 INFO:tasks.rados.rados.0.trial169.stdout:467: snap_remove snap 64 2026-02-25T08:43:59.096 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:43:59.166 INFO:tasks.rados.rados.0.trial169.stdout:467: done (6 left) 2026-02-25T08:43:59.167 INFO:tasks.rados.rados.0.trial169.stdout:468: write oid 29 current snap is 66 2026-02-25T08:43:59.167 INFO:tasks.rados.rados.0.trial169.stdout:468: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2026-02-25T08:43:59.168 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:43:59.170 INFO:tasks.rados.rados.0.trial169.stdout:468: writing trial16941936-29 from 682417 to 1155526 tid 1 2026-02-25T08:43:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:43:59.172 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:43:59.175 INFO:tasks.rados.rados.0.trial169.stdout:468: writing trial16941936-29 from 1567811 to 2169874 tid 2 2026-02-25T08:43:59.180 INFO:tasks.rados.rados.0.trial169.stdout:468: writing trial16941936-29 from 2844123 to 3465605 tid 3 2026-02-25T08:43:59.180 INFO:tasks.rados.rados.0.trial169.stdout:469: delete oid 17 current snap is 66 2026-02-25T08:43:59.180 INFO:tasks.rados.rados.0.trial169.stdout:461: expect (ObjNum 218 snap 63 seq_num 218) 2026-02-25T08:43:59.299 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1952156 2026-02-25T08:43:59.333 INFO:tasks.rados.rados.0.trial169.stdout:464: expect (ObjNum 149 snap 32 seq_num 149) 2026-02-25T08:43:59.369 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 294833 2026-02-25T08:43:59.388 INFO:tasks.rados.rados.0.trial169.stdout:460: expect (ObjNum 183 snap 46 seq_num 183) 2026-02-25T08:43:59.479 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2932444 2026-02-25T08:43:59.542 INFO:tasks.rados.rados.0.trial169.stdout:462: finishing write tid 1 to trial16941936-24 2026-02-25T08:43:59.542 INFO:tasks.rados.rados.0.trial169.stdout:462: finishing write tid 2 to trial16941936-24 2026-02-25T08:43:59.542 INFO:tasks.rados.rados.0.trial169.stdout:460: done (7 left) 2026-02-25T08:43:59.542 INFO:tasks.rados.rados.0.trial169.stdout:461: done (6 left) 2026-02-25T08:43:59.542 INFO:tasks.rados.rados.0.trial169.stdout:464: done (5 left) 2026-02-25T08:43:59.543 INFO:tasks.rados.rados.0.trial169.stdout:465: done (4 left) 2026-02-25T08:43:59.543 INFO:tasks.rados.rados.0.trial169.stdout:469: done (3 left) 2026-02-25T08:43:59.543 INFO:tasks.rados.rados.0.trial169.stdout:470: read oid 50 snap -1 2026-02-25T08:43:59.543 INFO:tasks.rados.rados.0.trial169.stdout:470: expect (ObjNum 189 snap 53 seq_num 189) 2026-02-25T08:43:59.543 INFO:tasks.rados.rados.0.trial169.stdout:471: read oid 46 snap -1 2026-02-25T08:43:59.543 INFO:tasks.rados.rados.0.trial169.stdout:471: expect (ObjNum 208 snap 60 seq_num 208) 2026-02-25T08:43:59.543 INFO:tasks.rados.rados.0.trial169.stdout:472: snap_remove snap 56 2026-02-25T08:43:59.908 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:43:59.986 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:59.983+0000 7fc177acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:59.988 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:43:59.987+0000 7fc177acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:43:59.992 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:00.031 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:00.032 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:00.172 INFO:tasks.rados.rados.0.trial169.stdout:472: done (5 left) 2026-02-25T08:44:00.173 INFO:tasks.rados.rados.0.trial169.stdout:473: read oid 6 snap -1 2026-02-25T08:44:00.173 INFO:tasks.rados.rados.0.trial169.stdout:473: expect (ObjNum 57 snap 1 seq_num 57) 2026-02-25T08:44:00.173 INFO:tasks.rados.rados.0.trial169.stdout:474: snap_create 2026-02-25T08:44:00.173 INFO:tasks.rados.rados.0.trial169.stdout:462: finishing write tid 3 to trial16941936-24 2026-02-25T08:44:00.173 INFO:tasks.rados.rados.0.trial169.stdout:462: finishing write tid 4 to trial16941936-24 2026-02-25T08:44:00.173 INFO:tasks.rados.rados.0.trial169.stdout:462: finishing write tid 5 to trial16941936-24 2026-02-25T08:44:00.173 INFO:tasks.rados.rados.0.trial169.stdout:462: oid 24 updating version 0 to 56 2026-02-25T08:44:00.173 INFO:tasks.rados.rados.0.trial169.stdout:462: oid 24 version 56 is already newer than 56 2026-02-25T08:44:00.173 INFO:tasks.rados.rados.0.trial169.stdout:462: oid 24 version 56 is already newer than 53 2026-02-25T08:44:00.173 INFO:tasks.rados.rados.0.trial169.stdout:462: oid 24 version 56 is already newer than 56 2026-02-25T08:44:00.174 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 24 v 56 (ObjNum 228 snap 66 seq_num 228) dirty exists 2026-02-25T08:44:00.174 INFO:tasks.rados.rados.0.trial169.stdout:462: left oid 24 (ObjNum 228 snap 66 seq_num 228) 2026-02-25T08:44:00.174 INFO:tasks.rados.rados.0.trial169.stdout:466: finishing write tid 1 to trial16941936-14 2026-02-25T08:44:00.174 INFO:tasks.rados.rados.0.trial169.stdout:466: finishing write tid 2 to trial16941936-14 2026-02-25T08:44:00.174 INFO:tasks.rados.rados.0.trial169.stdout:466: finishing write tid 3 to trial16941936-14 2026-02-25T08:44:00.174 INFO:tasks.rados.rados.0.trial169.stdout:466: finishing write tid 4 to trial16941936-14 2026-02-25T08:44:00.174 INFO:tasks.rados.rados.0.trial169.stdout:466: finishing write tid 5 to trial16941936-14 2026-02-25T08:44:00.174 INFO:tasks.rados.rados.0.trial169.stdout:466: oid 14 updating version 0 to 95 2026-02-25T08:44:00.174 INFO:tasks.rados.rados.0.trial169.stdout:466: oid 14 version 95 is already newer than 95 2026-02-25T08:44:00.174 INFO:tasks.rados.rados.0.trial169.stdout:466: oid 14 version 95 is already newer than 95 2026-02-25T08:44:00.174 INFO:tasks.rados.rados.0.trial169.stdout:466: oid 14 version 95 is already newer than 95 2026-02-25T08:44:00.174 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 14 v 95 (ObjNum 230 snap 66 seq_num 230) dirty exists 2026-02-25T08:44:00.174 INFO:tasks.rados.rados.0.trial169.stdout:466: left oid 14 (ObjNum 230 snap 66 seq_num 230) 2026-02-25T08:44:00.175 INFO:tasks.rados.rados.0.trial169.stdout:468: finishing write tid 1 to trial16941936-29 2026-02-25T08:44:00.175 INFO:tasks.rados.rados.0.trial169.stdout:468: finishing write tid 2 to trial16941936-29 2026-02-25T08:44:00.175 INFO:tasks.rados.rados.0.trial169.stdout:468: finishing write tid 3 to trial16941936-29 2026-02-25T08:44:00.175 INFO:tasks.rados.rados.0.trial169.stdout:468: finishing write tid 4 to trial16941936-29 2026-02-25T08:44:00.175 INFO:tasks.rados.rados.0.trial169.stdout:468: finishing write tid 5 to trial16941936-29 2026-02-25T08:44:00.175 INFO:tasks.rados.rados.0.trial169.stdout:468: oid 29 updating version 0 to 243 2026-02-25T08:44:00.175 INFO:tasks.rados.rados.0.trial169.stdout:468: oid 29 version 243 is already newer than 243 2026-02-25T08:44:00.175 INFO:tasks.rados.rados.0.trial169.stdout:468: oid 29 version 243 is already newer than 243 2026-02-25T08:44:00.175 INFO:tasks.rados.rados.0.trial169.stdout:468: oid 29 version 243 is already newer than 243 2026-02-25T08:44:00.175 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 29 v 243 (ObjNum 231 snap 66 seq_num 231) dirty exists 2026-02-25T08:44:00.175 INFO:tasks.rados.rados.0.trial169.stdout:468: left oid 29 (ObjNum 231 snap 66 seq_num 231) 2026-02-25T08:44:00.175 INFO:tasks.rados.rados.0.trial169.stdout:470: expect (ObjNum 189 snap 53 seq_num 189) 2026-02-25T08:44:00.193 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2198684 2026-02-25T08:44:00.281 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2232591 2026-02-25T08:44:00.293 INFO:tasks.rados.rados.0.trial169.stdout:471: expect (ObjNum 208 snap 60 seq_num 208) 2026-02-25T08:44:00.297 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1188982 2026-02-25T08:44:00.331 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:00.392 INFO:tasks.rados.rados.0.trial169.stdout:473: expect (ObjNum 57 snap 1 seq_num 57) 2026-02-25T08:44:00.472 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:01.175 INFO:tasks.rados.rados.0.trial169.stdout:462: done (6 left) 2026-02-25T08:44:01.175 INFO:tasks.rados.rados.0.trial169.stdout:466: done (5 left) 2026-02-25T08:44:01.175 INFO:tasks.rados.rados.0.trial169.stdout:468: done (4 left) 2026-02-25T08:44:01.175 INFO:tasks.rados.rados.0.trial169.stdout:470: done (3 left) 2026-02-25T08:44:01.175 INFO:tasks.rados.rados.0.trial169.stdout:471: done (2 left) 2026-02-25T08:44:01.176 INFO:tasks.rados.rados.0.trial169.stdout:473: done (1 left) 2026-02-25T08:44:01.176 INFO:tasks.rados.rados.0.trial169.stdout:474: done (0 left) 2026-02-25T08:44:01.176 INFO:tasks.rados.rados.0.trial169.stdout:475: read oid 45 snap -1 2026-02-25T08:44:01.176 INFO:tasks.rados.rados.0.trial169.stdout:475: expect (ObjNum 173 snap 43 seq_num 173) 2026-02-25T08:44:01.176 INFO:tasks.rados.rados.0.trial169.stdout:476: snap_remove snap 59 2026-02-25T08:44:01.599 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~116s 2026-02-25T08:44:02.032 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:02.109 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:02.107+0000 7fb3ac65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:02.112 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:02.110+0000 7fb3ab3d1640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:02.115 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:02.155 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:02.177 INFO:tasks.rados.rados.0.trial169.stdout:476: done (1 left) 2026-02-25T08:44:02.177 INFO:tasks.rados.rados.0.trial169.stdout:477: delete oid 38 current snap is 67 2026-02-25T08:44:02.177 INFO:tasks.rados.rados.0.trial169.stdout:475: expect (ObjNum 173 snap 43 seq_num 173) 2026-02-25T08:44:02.245 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2313776 2026-02-25T08:44:02.296 INFO:tasks.rados.rados.0.trial169.stdout:475: done (1 left) 2026-02-25T08:44:02.296 INFO:tasks.rados.rados.0.trial169.stdout:477: done (0 left) 2026-02-25T08:44:02.296 INFO:tasks.rados.rados.0.trial169.stdout:478: write (excl) oid 20 current snap is 67 2026-02-25T08:44:02.296 INFO:tasks.rados.rados.0.trial169.stdout:478: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2026-02-25T08:44:02.301 INFO:tasks.rados.rados.0.trial169.stdout:478: writing trial16941936-20 from 732838 to 1431694 tid 1 2026-02-25T08:44:02.306 INFO:tasks.rados.rados.0.trial169.stdout:478: writing trial16941936-20 from 2026904 to 2492647 tid 2 2026-02-25T08:44:02.310 INFO:tasks.rados.rados.0.trial169.stdout:478: writing trial16941936-20 from 3210039 to 3781690 tid 3 2026-02-25T08:44:02.311 INFO:tasks.rados.rados.0.trial169.stdout:479: snap_remove snap 62 2026-02-25T08:44:03.179 INFO:tasks.rados.rados.0.trial169.stdout:479: done (1 left) 2026-02-25T08:44:03.180 INFO:tasks.rados.rados.0.trial169.stdout:480: delete oid 36 current snap is 67 2026-02-25T08:44:03.180 INFO:tasks.rados.rados.0.trial169.stdout:478: finishing write tid 1 to trial16941936-20 2026-02-25T08:44:03.180 INFO:tasks.rados.rados.0.trial169.stdout:478: finishing write tid 2 to trial16941936-20 2026-02-25T08:44:03.180 INFO:tasks.rados.rados.0.trial169.stdout:478: finishing write tid 3 to trial16941936-20 2026-02-25T08:44:03.180 INFO:tasks.rados.rados.0.trial169.stdout:478: finishing write tid 4 to trial16941936-20 2026-02-25T08:44:03.180 INFO:tasks.rados.rados.0.trial169.stdout:478: finishing write tid 5 to trial16941936-20 2026-02-25T08:44:03.180 INFO:tasks.rados.rados.0.trial169.stdout:478: oid 20 updating version 0 to 90 2026-02-25T08:44:03.180 INFO:tasks.rados.rados.0.trial169.stdout:478: oid 20 updating version 90 to 93 2026-02-25T08:44:03.180 INFO:tasks.rados.rados.0.trial169.stdout:478: oid 20 version 93 is already newer than 93 2026-02-25T08:44:03.180 INFO:tasks.rados.rados.0.trial169.stdout:478: oid 20 version 93 is already newer than 93 2026-02-25T08:44:03.181 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 20 v 93 (ObjNum 234 snap 67 seq_num 234) dirty exists 2026-02-25T08:44:03.181 INFO:tasks.rados.rados.0.trial169.stdout:478: left oid 20 (ObjNum 234 snap 67 seq_num 234) 2026-02-25T08:44:03.222 INFO:tasks.rados.rados.0.trial169.stdout:478: done (1 left) 2026-02-25T08:44:03.223 INFO:tasks.rados.rados.0.trial169.stdout:480: done (0 left) 2026-02-25T08:44:03.223 INFO:tasks.rados.rados.0.trial169.stdout:481: delete oid 1 current snap is 67 2026-02-25T08:44:03.225 INFO:tasks.rados.rados.0.trial169.stdout:481: done (0 left) 2026-02-25T08:44:03.225 INFO:tasks.rados.rados.0.trial169.stdout:482: read oid 17 snap -1 2026-02-25T08:44:03.225 INFO:tasks.rados.rados.0.trial169.stdout:482: expect deleted 2026-02-25T08:44:03.225 INFO:tasks.rados.rados.0.trial169.stdout:483: read oid 43 snap -1 2026-02-25T08:44:03.225 INFO:tasks.rados.rados.0.trial169.stdout:483: expect (ObjNum 177 snap 46 seq_num 177) 2026-02-25T08:44:03.226 INFO:tasks.rados.rados.0.trial169.stdout:484: read oid 37 snap 46 2026-02-25T08:44:03.226 INFO:tasks.rados.rados.0.trial169.stdout:484: expect (ObjNum 129 snap 23 seq_num 129) 2026-02-25T08:44:03.226 INFO:tasks.rados.rados.0.trial169.stdout:485: snap_remove snap 54 2026-02-25T08:44:04.157 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:04.172 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:44:04.182 INFO:tasks.rados.rados.0.trial169.stdout:485: done (3 left) 2026-02-25T08:44:04.183 INFO:tasks.rados.rados.0.trial169.stdout:486: delete oid 7 current snap is 67 2026-02-25T08:44:04.183 INFO:tasks.rados.rados.0.trial169.stdout:483: expect (ObjNum 177 snap 46 seq_num 177) 2026-02-25T08:44:04.183 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:04.236 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:04.234+0000 7f8a6f79c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:04.239 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:04.238+0000 7f8a6f79c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:04.243 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:04.252 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:04.255 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:44:04.300 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:04.301 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:04.304 INFO:tasks.rados.rados.0.trial169.stdout:484: expect (ObjNum 129 snap 23 seq_num 129) 2026-02-25T08:44:04.308 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 609087 2026-02-25T08:44:04.357 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 311360 2026-02-25T08:44:04.388 INFO:tasks.rados.rados.0.trial169.stdout:482: done (3 left) 2026-02-25T08:44:04.388 INFO:tasks.rados.rados.0.trial169.stdout:483: done (2 left) 2026-02-25T08:44:04.388 INFO:tasks.rados.rados.0.trial169.stdout:484: done (1 left) 2026-02-25T08:44:04.388 INFO:tasks.rados.rados.0.trial169.stdout:486: done (0 left) 2026-02-25T08:44:04.388 INFO:tasks.rados.rados.0.trial169.stdout:487: write oid 19 current snap is 67 2026-02-25T08:44:04.389 INFO:tasks.rados.rados.0.trial169.stdout:487: seq_num 238 ranges {400066=716205,1860832=139234} 2026-02-25T08:44:04.392 INFO:tasks.rados.rados.0.trial169.stdout:487: writing trial16941936-19 from 400066 to 1116271 tid 1 2026-02-25T08:44:04.396 INFO:tasks.rados.rados.0.trial169.stdout:487: writing trial16941936-19 from 1860832 to 2000066 tid 2 2026-02-25T08:44:04.396 INFO:tasks.rados.rados.0.trial169.stdout:488: read oid 45 snap -1 2026-02-25T08:44:04.396 INFO:tasks.rados.rados.0.trial169.stdout:488: expect (ObjNum 173 snap 43 seq_num 173) 2026-02-25T08:44:04.396 INFO:tasks.rados.rados.0.trial169.stdout:489: snap_create 2026-02-25T08:44:04.413 INFO:tasks.rados.rados.0.trial169.stdout:487: finishing write tid 1 to trial16941936-19 2026-02-25T08:44:04.422 INFO:tasks.rados.rados.0.trial169.stdout:487: finishing write tid 2 to trial16941936-19 2026-02-25T08:44:04.423 INFO:tasks.rados.rados.0.trial169.stdout:487: finishing write tid 3 to trial16941936-19 2026-02-25T08:44:04.423 INFO:tasks.rados.rados.0.trial169.stdout:488: expect (ObjNum 173 snap 43 seq_num 173) 2026-02-25T08:44:04.508 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:04.552 INFO:tasks.rados.rados.0.trial169.stdout:487: finishing write tid 4 to trial16941936-19 2026-02-25T08:44:04.552 INFO:tasks.rados.rados.0.trial169.stdout:487: oid 19 updating version 0 to 65 2026-02-25T08:44:04.552 INFO:tasks.rados.rados.0.trial169.stdout:487: oid 19 version 65 is already newer than 65 2026-02-25T08:44:04.552 INFO:tasks.rados.rados.0.trial169.stdout:487: oid 19 version 65 is already newer than 65 2026-02-25T08:44:04.552 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 19 v 65 (ObjNum 238 snap 67 seq_num 238) dirty exists 2026-02-25T08:44:04.552 INFO:tasks.rados.rados.0.trial169.stdout:487: left oid 19 (ObjNum 238 snap 67 seq_num 238) 2026-02-25T08:44:05.186 INFO:tasks.rados.rados.0.trial169.stdout:487: done (2 left) 2026-02-25T08:44:05.186 INFO:tasks.rados.rados.0.trial169.stdout:488: done (1 left) 2026-02-25T08:44:05.186 INFO:tasks.rados.rados.0.trial169.stdout:489: done (0 left) 2026-02-25T08:44:05.186 INFO:tasks.rados.rados.0.trial169.stdout:490: write (excl) oid 24 current snap is 68 2026-02-25T08:44:05.187 INFO:tasks.rados.rados.0.trial169.stdout:490: seq_num 239 ranges {416873=791454,1919314=97559} 2026-02-25T08:44:05.191 INFO:tasks.rados.rados.0.trial169.stdout:490: writing trial16941936-24 from 416873 to 1208327 tid 1 2026-02-25T08:44:05.195 INFO:tasks.rados.rados.0.trial169.stdout:490: writing trial16941936-24 from 1919314 to 2016873 tid 2 2026-02-25T08:44:05.195 INFO:tasks.rados.rados.0.trial169.stdout:491: read oid 4 snap -1 2026-02-25T08:44:05.195 INFO:tasks.rados.rados.0.trial169.stdout:491: expect deleted 2026-02-25T08:44:05.195 INFO:tasks.rados.rados.0.trial169.stdout:492: read oid 6 snap -1 2026-02-25T08:44:05.195 INFO:tasks.rados.rados.0.trial169.stdout:492: expect (ObjNum 57 snap 1 seq_num 57) 2026-02-25T08:44:05.195 INFO:tasks.rados.rados.0.trial169.stdout:493: read oid 41 snap 65 2026-02-25T08:44:05.195 INFO:tasks.rados.rados.0.trial169.stdout:493: expect (ObjNum 137 snap 26 seq_num 137) 2026-02-25T08:44:05.195 INFO:tasks.rados.rados.0.trial169.stdout:494: delete oid 8 current snap is 68 2026-02-25T08:44:05.211 INFO:tasks.rados.rados.0.trial169.stdout:492: expect (ObjNum 57 snap 1 seq_num 57) 2026-02-25T08:44:05.211 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:05.245 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:05.303 INFO:tasks.rados.rados.0.trial169.stdout:491: done (4 left) 2026-02-25T08:44:05.303 INFO:tasks.rados.rados.0.trial169.stdout:492: done (3 left) 2026-02-25T08:44:05.303 INFO:tasks.rados.rados.0.trial169.stdout:494: done (2 left) 2026-02-25T08:44:05.303 INFO:tasks.rados.rados.0.trial169.stdout:495: snap_remove snap 46 2026-02-25T08:44:06.190 INFO:tasks.rados.rados.0.trial169.stdout:495: done (2 left) 2026-02-25T08:44:06.190 INFO:tasks.rados.rados.0.trial169.stdout:496: snap_create 2026-02-25T08:44:06.190 INFO:tasks.rados.rados.0.trial169.stdout:493: expect (ObjNum 137 snap 26 seq_num 137) 2026-02-25T08:44:06.190 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:06.237 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1798673 2026-02-25T08:44:06.244 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:06.276 INFO:tasks.rados.rados.0.trial169.stdout:490: finishing write tid 1 to trial16941936-24 2026-02-25T08:44:06.276 INFO:tasks.rados.rados.0.trial169.stdout:490: finishing write tid 2 to trial16941936-24 2026-02-25T08:44:06.276 INFO:tasks.rados.rados.0.trial169.stdout:490: finishing write tid 3 to trial16941936-24 2026-02-25T08:44:06.277 INFO:tasks.rados.rados.0.trial169.stdout:490: finishing write tid 4 to trial16941936-24 2026-02-25T08:44:06.277 INFO:tasks.rados.rados.0.trial169.stdout:490: oid 24 updating version 0 to 60 2026-02-25T08:44:06.277 INFO:tasks.rados.rados.0.trial169.stdout:490: oid 24 version 60 is already newer than 60 2026-02-25T08:44:06.277 INFO:tasks.rados.rados.0.trial169.stdout:490: oid 24 version 60 is already newer than 58 2026-02-25T08:44:06.277 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 24 v 60 (ObjNum 239 snap 68 seq_num 239) dirty exists 2026-02-25T08:44:06.277 INFO:tasks.rados.rados.0.trial169.stdout:490: left oid 24 (ObjNum 239 snap 68 seq_num 239) 2026-02-25T08:44:06.301 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:06.378 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:06.375+0000 7f88d0acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:06.380 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:06.379+0000 7f88d0acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:06.384 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:06.426 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:06.427 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:07.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~121s 2026-02-25T08:44:07.191 INFO:tasks.rados.rados.0.trial169.stdout:490: done (2 left) 2026-02-25T08:44:07.191 INFO:tasks.rados.rados.0.trial169.stdout:493: done (1 left) 2026-02-25T08:44:07.191 INFO:tasks.rados.rados.0.trial169.stdout:496: done (0 left) 2026-02-25T08:44:07.191 INFO:tasks.rados.rados.0.trial169.stdout:497: read oid 19 snap -1 2026-02-25T08:44:07.191 INFO:tasks.rados.rados.0.trial169.stdout:497: expect (ObjNum 238 snap 67 seq_num 238) 2026-02-25T08:44:07.192 INFO:tasks.rados.rados.0.trial169.stdout:498: delete oid 3 current snap is 69 2026-02-25T08:44:07.217 INFO:tasks.rados.rados.0.trial169.stdout:497: expect (ObjNum 238 snap 67 seq_num 238) 2026-02-25T08:44:07.217 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 152097 2026-02-25T08:44:07.265 INFO:tasks.rados.rados.0.trial169.stdout:497: done (1 left) 2026-02-25T08:44:07.265 INFO:tasks.rados.rados.0.trial169.stdout:498: done (0 left) 2026-02-25T08:44:07.265 INFO:tasks.rados.rados.0.trial169.stdout:499: read oid 22 snap 63 2026-02-25T08:44:07.265 INFO:tasks.rados.rados.0.trial169.stdout:499: expect deleted 2026-02-25T08:44:07.265 INFO:tasks.rados.rados.0.trial169.stdout:500: read oid 12 snap -1 2026-02-25T08:44:07.265 INFO:tasks.rados.rados.0.trial169.stdout:500: expect deleted 2026-02-25T08:44:07.265 INFO:tasks.rados.rados.0.trial169.stdout:501: write (excl) oid 18 current snap is 69 2026-02-25T08:44:07.266 INFO:tasks.rados.rados.0.trial169.stdout:501: seq_num 242 ranges {467294=617201,1578407=488887} 2026-02-25T08:44:07.268 INFO:tasks.rados.rados.0.trial169.stdout:501: writing trial16941936-18 from 467294 to 1084495 tid 1 2026-02-25T08:44:07.273 INFO:tasks.rados.rados.0.trial169.stdout:501: writing trial16941936-18 from 1578407 to 2067294 tid 2 2026-02-25T08:44:07.273 INFO:tasks.rados.rados.0.trial169.stdout:502: read oid 45 snap -1 2026-02-25T08:44:07.273 INFO:tasks.rados.rados.0.trial169.stdout:502: expect (ObjNum 173 snap 43 seq_num 173) 2026-02-25T08:44:07.273 INFO:tasks.rados.rados.0.trial169.stdout:503: read oid 31 snap -1 2026-02-25T08:44:07.273 INFO:tasks.rados.rados.0.trial169.stdout:503: expect deleted 2026-02-25T08:44:07.273 INFO:tasks.rados.rados.0.trial169.stdout:504: snap_remove snap 65 2026-02-25T08:44:08.136 INFO:tasks.rados.rados.0.trial169.stdout:504: done (5 left) 2026-02-25T08:44:08.136 INFO:tasks.rados.rados.0.trial169.stdout:505: delete oid 47 current snap is 69 2026-02-25T08:44:08.136 INFO:tasks.rados.rados.0.trial169.stdout:501: finishing write tid 1 to trial16941936-18 2026-02-25T08:44:08.136 INFO:tasks.rados.rados.0.trial169.stdout:502: expect (ObjNum 173 snap 43 seq_num 173) 2026-02-25T08:44:08.136 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:08.178 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:08.265 INFO:tasks.rados.rados.0.trial169.stdout:501: finishing write tid 2 to trial16941936-18 2026-02-25T08:44:08.266 INFO:tasks.rados.rados.0.trial169.stdout:501: finishing write tid 3 to trial16941936-18 2026-02-25T08:44:08.266 INFO:tasks.rados.rados.0.trial169.stdout:501: finishing write tid 4 to trial16941936-18 2026-02-25T08:44:08.266 INFO:tasks.rados.rados.0.trial169.stdout:501: oid 18 updating version 0 to 82 2026-02-25T08:44:08.266 INFO:tasks.rados.rados.0.trial169.stdout:501: oid 18 version 82 is already newer than 80 2026-02-25T08:44:08.266 INFO:tasks.rados.rados.0.trial169.stdout:501: oid 18 version 82 is already newer than 82 2026-02-25T08:44:08.266 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 18 v 82 (ObjNum 242 snap 69 seq_num 242) dirty exists 2026-02-25T08:44:08.266 INFO:tasks.rados.rados.0.trial169.stdout:501: left oid 18 (ObjNum 242 snap 69 seq_num 242) 2026-02-25T08:44:08.266 INFO:tasks.rados.rados.0.trial169.stdout:499: done (5 left) 2026-02-25T08:44:08.266 INFO:tasks.rados.rados.0.trial169.stdout:500: done (4 left) 2026-02-25T08:44:08.266 INFO:tasks.rados.rados.0.trial169.stdout:501: done (3 left) 2026-02-25T08:44:08.266 INFO:tasks.rados.rados.0.trial169.stdout:502: done (2 left) 2026-02-25T08:44:08.267 INFO:tasks.rados.rados.0.trial169.stdout:503: done (1 left) 2026-02-25T08:44:08.267 INFO:tasks.rados.rados.0.trial169.stdout:505: done (0 left) 2026-02-25T08:44:08.267 INFO:tasks.rados.rados.0.trial169.stdout:506: read oid 22 snap -1 2026-02-25T08:44:08.267 INFO:tasks.rados.rados.0.trial169.stdout:506: expect deleted 2026-02-25T08:44:08.267 INFO:tasks.rados.rados.0.trial169.stdout:507: read oid 12 snap -1 2026-02-25T08:44:08.267 INFO:tasks.rados.rados.0.trial169.stdout:507: expect deleted 2026-02-25T08:44:08.267 INFO:tasks.rados.rados.0.trial169.stdout:508: snap_remove snap 63 2026-02-25T08:44:08.428 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:08.505 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:08.503+0000 7f787efa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:08.508 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:08.506+0000 7f787efa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:08.511 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:08.550 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:08.551 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:09.200 INFO:tasks.rados.rados.0.trial169.stdout:508: done (2 left) 2026-02-25T08:44:09.200 INFO:tasks.rados.rados.0.trial169.stdout:509: write oid 50 current snap is 69 2026-02-25T08:44:09.200 INFO:tasks.rados.rados.0.trial169.stdout:509: seq_num 244 ranges {500908=484052,1411724=689184} 2026-02-25T08:44:09.203 INFO:tasks.rados.rados.0.trial169.stdout:509: writing trial16941936-50 from 500908 to 984960 tid 1 2026-02-25T08:44:09.208 INFO:tasks.rados.rados.0.trial169.stdout:509: writing trial16941936-50 from 1411724 to 2100908 tid 2 2026-02-25T08:44:09.208 INFO:tasks.rados.rados.0.trial169.stdout:510: snap_remove snap 51 2026-02-25T08:44:09.256 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:44:09.327 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:09.330 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:09.330 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:44:10.204 INFO:tasks.rados.rados.0.trial169.stdout:510: done (3 left) 2026-02-25T08:44:10.204 INFO:tasks.rados.rados.0.trial169.stdout:511: write (excl) oid 25 current snap is 69 2026-02-25T08:44:10.204 INFO:tasks.rados.rados.0.trial169.stdout:511: seq_num 245 ranges {517715=559301,1870206=247509} 2026-02-25T08:44:10.207 INFO:tasks.rados.rados.0.trial169.stdout:511: writing trial16941936-25 from 517715 to 1077016 tid 1 2026-02-25T08:44:10.212 INFO:tasks.rados.rados.0.trial169.stdout:511: writing trial16941936-25 from 1870206 to 2117715 tid 2 2026-02-25T08:44:10.212 INFO:tasks.rados.rados.0.trial169.stdout:512: write oid 47 current snap is 69 2026-02-25T08:44:10.212 INFO:tasks.rados.rados.0.trial169.stdout:512: seq_num 246 ranges {534522=634550,1812335=322187} 2026-02-25T08:44:10.216 INFO:tasks.rados.rados.0.trial169.stdout:512: writing trial16941936-47 from 534522 to 1169072 tid 1 2026-02-25T08:44:10.220 INFO:tasks.rados.rados.0.trial169.stdout:512: writing trial16941936-47 from 1812335 to 2134522 tid 2 2026-02-25T08:44:10.220 INFO:tasks.rados.rados.0.trial169.stdout:513: read oid 18 snap 60 2026-02-25T08:44:10.221 INFO:tasks.rados.rados.0.trial169.stdout:513: expect (ObjNum 196 snap 54 seq_num 196) 2026-02-25T08:44:10.221 INFO:tasks.rados.rados.0.trial169.stdout:514: snap_remove snap 67 2026-02-25T08:44:10.552 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:10.630 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:10.627+0000 7ff6d95b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:10.632 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:10.630+0000 7ff6d95b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:10.635 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:10.674 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:11.207 INFO:tasks.rados.rados.0.trial169.stdout:514: done (6 left) 2026-02-25T08:44:11.208 INFO:tasks.rados.rados.0.trial169.stdout:515: snap_create 2026-02-25T08:44:11.208 INFO:tasks.rados.rados.0.trial169.stdout:509: finishing write tid 1 to trial16941936-50 2026-02-25T08:44:11.208 INFO:tasks.rados.rados.0.trial169.stdout:509: finishing write tid 2 to trial16941936-50 2026-02-25T08:44:11.208 INFO:tasks.rados.rados.0.trial169.stdout:509: finishing write tid 3 to trial16941936-50 2026-02-25T08:44:11.208 INFO:tasks.rados.rados.0.trial169.stdout:509: finishing write tid 4 to trial16941936-50 2026-02-25T08:44:11.208 INFO:tasks.rados.rados.0.trial169.stdout:509: oid 50 updating version 0 to 263 2026-02-25T08:44:11.208 INFO:tasks.rados.rados.0.trial169.stdout:509: oid 50 version 263 is already newer than 263 2026-02-25T08:44:11.208 INFO:tasks.rados.rados.0.trial169.stdout:509: oid 50 version 263 is already newer than 263 2026-02-25T08:44:11.208 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 50 v 263 (ObjNum 244 snap 69 seq_num 244) dirty exists 2026-02-25T08:44:11.209 INFO:tasks.rados.rados.0.trial169.stdout:509: left oid 50 (ObjNum 244 snap 69 seq_num 244) 2026-02-25T08:44:11.209 INFO:tasks.rados.rados.0.trial169.stdout:511: finishing write tid 1 to trial16941936-25 2026-02-25T08:44:11.209 INFO:tasks.rados.rados.0.trial169.stdout:513: expect (ObjNum 196 snap 54 seq_num 196) 2026-02-25T08:44:11.209 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:11.253 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:11.343 INFO:tasks.rados.rados.0.trial169.stdout:511: finishing write tid 2 to trial16941936-25 2026-02-25T08:44:11.343 INFO:tasks.rados.rados.0.trial169.stdout:511: finishing write tid 3 to trial16941936-25 2026-02-25T08:44:11.343 INFO:tasks.rados.rados.0.trial169.stdout:511: finishing write tid 4 to trial16941936-25 2026-02-25T08:44:11.343 INFO:tasks.rados.rados.0.trial169.stdout:511: oid 25 updating version 0 to 44 2026-02-25T08:44:11.343 INFO:tasks.rados.rados.0.trial169.stdout:511: oid 25 updating version 44 to 46 2026-02-25T08:44:11.343 INFO:tasks.rados.rados.0.trial169.stdout:511: oid 25 version 46 is already newer than 46 2026-02-25T08:44:11.344 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 25 v 46 (ObjNum 245 snap 69 seq_num 245) dirty exists 2026-02-25T08:44:11.344 INFO:tasks.rados.rados.0.trial169.stdout:511: left oid 25 (ObjNum 245 snap 69 seq_num 245) 2026-02-25T08:44:11.344 INFO:tasks.rados.rados.0.trial169.stdout:512: finishing write tid 1 to trial16941936-47 2026-02-25T08:44:11.344 INFO:tasks.rados.rados.0.trial169.stdout:512: finishing write tid 2 to trial16941936-47 2026-02-25T08:44:11.344 INFO:tasks.rados.rados.0.trial169.stdout:512: finishing write tid 3 to trial16941936-47 2026-02-25T08:44:11.344 INFO:tasks.rados.rados.0.trial169.stdout:512: finishing write tid 4 to trial16941936-47 2026-02-25T08:44:11.344 INFO:tasks.rados.rados.0.trial169.stdout:512: oid 47 updating version 0 to 49 2026-02-25T08:44:11.344 INFO:tasks.rados.rados.0.trial169.stdout:512: oid 47 version 49 is already newer than 49 2026-02-25T08:44:11.344 INFO:tasks.rados.rados.0.trial169.stdout:512: oid 47 version 49 is already newer than 49 2026-02-25T08:44:11.344 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 47 v 49 (ObjNum 246 snap 69 seq_num 246) dirty exists 2026-02-25T08:44:11.344 INFO:tasks.rados.rados.0.trial169.stdout:512: left oid 47 (ObjNum 246 snap 69 seq_num 246) 2026-02-25T08:44:12.210 INFO:tasks.rados.rados.0.trial169.stdout:506: done (6 left) 2026-02-25T08:44:12.210 INFO:tasks.rados.rados.0.trial169.stdout:507: done (5 left) 2026-02-25T08:44:12.210 INFO:tasks.rados.rados.0.trial169.stdout:509: done (4 left) 2026-02-25T08:44:12.210 INFO:tasks.rados.rados.0.trial169.stdout:511: done (3 left) 2026-02-25T08:44:12.211 INFO:tasks.rados.rados.0.trial169.stdout:512: done (2 left) 2026-02-25T08:44:12.211 INFO:tasks.rados.rados.0.trial169.stdout:513: done (1 left) 2026-02-25T08:44:12.211 INFO:tasks.rados.rados.0.trial169.stdout:515: done (0 left) 2026-02-25T08:44:12.211 INFO:tasks.rados.rados.0.trial169.stdout:516: delete oid 2 current snap is 70 2026-02-25T08:44:12.251 INFO:tasks.rados.rados.0.trial169.stdout:516: done (0 left) 2026-02-25T08:44:12.252 INFO:tasks.rados.rados.0.trial169.stdout:517: read oid 14 snap -1 2026-02-25T08:44:12.252 INFO:tasks.rados.rados.0.trial169.stdout:517: expect (ObjNum 230 snap 66 seq_num 230) 2026-02-25T08:44:12.252 INFO:tasks.rados.rados.0.trial169.stdout:518: write oid 31 current snap is 70 2026-02-25T08:44:12.252 INFO:tasks.rados.rados.0.trial169.stdout:518: seq_num 248 ranges {568136=785048,1929299=238837} 2026-02-25T08:44:12.256 INFO:tasks.rados.rados.0.trial169.stdout:518: writing trial16941936-31 from 568136 to 1353184 tid 1 2026-02-25T08:44:12.260 INFO:tasks.rados.rados.0.trial169.stdout:518: writing trial16941936-31 from 1929299 to 2168136 tid 2 2026-02-25T08:44:12.260 INFO:tasks.rados.rados.0.trial169.stdout:519: snap_create 2026-02-25T08:44:12.280 INFO:tasks.rados.rados.0.trial169.stdout:518: finishing write tid 1 to trial16941936-31 2026-02-25T08:44:12.285 INFO:tasks.rados.rados.0.trial169.stdout:517: expect (ObjNum 230 snap 66 seq_num 230) 2026-02-25T08:44:12.338 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:12.392 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:12.445 INFO:tasks.rados.rados.0.trial169.stdout:518: finishing write tid 2 to trial16941936-31 2026-02-25T08:44:12.445 INFO:tasks.rados.rados.0.trial169.stdout:518: finishing write tid 3 to trial16941936-31 2026-02-25T08:44:12.445 INFO:tasks.rados.rados.0.trial169.stdout:518: finishing write tid 4 to trial16941936-31 2026-02-25T08:44:12.445 INFO:tasks.rados.rados.0.trial169.stdout:518: oid 31 updating version 0 to 198 2026-02-25T08:44:12.445 INFO:tasks.rados.rados.0.trial169.stdout:518: oid 31 version 198 is already newer than 198 2026-02-25T08:44:12.446 INFO:tasks.rados.rados.0.trial169.stdout:518: oid 31 version 198 is already newer than 198 2026-02-25T08:44:12.446 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 31 v 198 (ObjNum 248 snap 70 seq_num 248) dirty exists 2026-02-25T08:44:12.446 INFO:tasks.rados.rados.0.trial169.stdout:518: left oid 31 (ObjNum 248 snap 70 seq_num 248) 2026-02-25T08:44:12.606 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~127s 2026-02-25T08:44:12.675 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:12.754 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:12.752+0000 7f68283ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:12.757 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:12.755+0000 7f68283ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:12.760 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:12.799 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:12.800 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:13.215 INFO:tasks.rados.rados.0.trial169.stdout:517: done (2 left) 2026-02-25T08:44:13.216 INFO:tasks.rados.rados.0.trial169.stdout:518: done (1 left) 2026-02-25T08:44:13.216 INFO:tasks.rados.rados.0.trial169.stdout:519: done (0 left) 2026-02-25T08:44:13.216 INFO:tasks.rados.rados.0.trial169.stdout:520: read oid 15 snap 50 2026-02-25T08:44:13.216 INFO:tasks.rados.rados.0.trial169.stdout:520: expect deleted 2026-02-25T08:44:13.216 INFO:tasks.rados.rados.0.trial169.stdout:521: read oid 43 snap 68 2026-02-25T08:44:13.216 INFO:tasks.rados.rados.0.trial169.stdout:521: expect (ObjNum 177 snap 46 seq_num 177) 2026-02-25T08:44:13.216 INFO:tasks.rados.rados.0.trial169.stdout:522: write oid 16 current snap is 71 2026-02-25T08:44:13.216 INFO:tasks.rados.rados.0.trial169.stdout:522: seq_num 249 ranges {584943=460297,1587781=597162} 2026-02-25T08:44:13.219 INFO:tasks.rados.rados.0.trial169.stdout:522: writing trial16941936-16 from 584943 to 1045240 tid 1 2026-02-25T08:44:13.223 INFO:tasks.rados.rados.0.trial169.stdout:522: writing trial16941936-16 from 1587781 to 2184943 tid 2 2026-02-25T08:44:13.223 INFO:tasks.rados.rados.0.trial169.stdout:523: delete oid 2 current snap is 71 2026-02-25T08:44:13.235 INFO:tasks.rados.rados.0.trial169.stdout:523: done (3 left) 2026-02-25T08:44:13.235 INFO:tasks.rados.rados.0.trial169.stdout:524: snap_remove snap 60 2026-02-25T08:44:14.219 INFO:tasks.rados.rados.0.trial169.stdout:524: done (3 left) 2026-02-25T08:44:14.219 INFO:tasks.rados.rados.0.trial169.stdout:525: read oid 26 snap -1 2026-02-25T08:44:14.220 INFO:tasks.rados.rados.0.trial169.stdout:525: expect (ObjNum 216 snap 63 seq_num 216) 2026-02-25T08:44:14.220 INFO:tasks.rados.rados.0.trial169.stdout:526: snap_create 2026-02-25T08:44:14.220 INFO:tasks.rados.rados.0.trial169.stdout:521: expect (ObjNum 177 snap 46 seq_num 177) 2026-02-25T08:44:14.271 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2030769 2026-02-25T08:44:14.331 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:44:14.337 INFO:tasks.rados.rados.0.trial169.stdout:522: finishing write tid 1 to trial16941936-16 2026-02-25T08:44:14.338 INFO:tasks.rados.rados.0.trial169.stdout:522: finishing write tid 2 to trial16941936-16 2026-02-25T08:44:14.338 INFO:tasks.rados.rados.0.trial169.stdout:522: finishing write tid 3 to trial16941936-16 2026-02-25T08:44:14.338 INFO:tasks.rados.rados.0.trial169.stdout:522: finishing write tid 4 to trial16941936-16 2026-02-25T08:44:14.338 INFO:tasks.rados.rados.0.trial169.stdout:522: oid 16 updating version 0 to 116 2026-02-25T08:44:14.338 INFO:tasks.rados.rados.0.trial169.stdout:522: oid 16 version 116 is already newer than 116 2026-02-25T08:44:14.338 INFO:tasks.rados.rados.0.trial169.stdout:522: oid 16 version 116 is already newer than 116 2026-02-25T08:44:14.338 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 16 v 116 (ObjNum 249 snap 71 seq_num 249) dirty exists 2026-02-25T08:44:14.338 INFO:tasks.rados.rados.0.trial169.stdout:522: left oid 16 (ObjNum 249 snap 71 seq_num 249) 2026-02-25T08:44:14.338 INFO:tasks.rados.rados.0.trial169.stdout:525: expect (ObjNum 216 snap 63 seq_num 216) 2026-02-25T08:44:14.338 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:14.401 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:14.404 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:14.405 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:44:14.801 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:14.877 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:14.874+0000 7f15b4e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:14.879 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:14.878+0000 7f15b4e1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:14.883 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:14.925 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:15.220 INFO:tasks.rados.rados.0.trial169.stdout:520: done (4 left) 2026-02-25T08:44:15.220 INFO:tasks.rados.rados.0.trial169.stdout:521: done (3 left) 2026-02-25T08:44:15.220 INFO:tasks.rados.rados.0.trial169.stdout:522: done (2 left) 2026-02-25T08:44:15.220 INFO:tasks.rados.rados.0.trial169.stdout:525: done (1 left) 2026-02-25T08:44:15.220 INFO:tasks.rados.rados.0.trial169.stdout:526: done (0 left) 2026-02-25T08:44:15.220 INFO:tasks.rados.rados.0.trial169.stdout:527: write oid 12 current snap is 72 2026-02-25T08:44:15.220 INFO:tasks.rados.rados.0.trial169.stdout:527: seq_num 251 ranges {618557=727148,2104745=113812} 2026-02-25T08:44:15.224 INFO:tasks.rados.rados.0.trial169.stdout:527: writing trial16941936-12 from 618557 to 1345705 tid 1 2026-02-25T08:44:15.228 INFO:tasks.rados.rados.0.trial169.stdout:527: writing trial16941936-12 from 2104745 to 2218557 tid 2 2026-02-25T08:44:15.228 INFO:tasks.rados.rados.0.trial169.stdout:528: snap_remove snap 55 2026-02-25T08:44:16.226 INFO:tasks.rados.rados.0.trial169.stdout:528: done (1 left) 2026-02-25T08:44:16.226 INFO:tasks.rados.rados.0.trial169.stdout:529: delete oid 31 current snap is 72 2026-02-25T08:44:16.226 INFO:tasks.rados.rados.0.trial169.stdout:527: finishing write tid 1 to trial16941936-12 2026-02-25T08:44:16.226 INFO:tasks.rados.rados.0.trial169.stdout:527: finishing write tid 2 to trial16941936-12 2026-02-25T08:44:16.226 INFO:tasks.rados.rados.0.trial169.stdout:527: finishing write tid 3 to trial16941936-12 2026-02-25T08:44:16.226 INFO:tasks.rados.rados.0.trial169.stdout:527: finishing write tid 4 to trial16941936-12 2026-02-25T08:44:16.226 INFO:tasks.rados.rados.0.trial169.stdout:527: oid 12 updating version 0 to 63 2026-02-25T08:44:16.226 INFO:tasks.rados.rados.0.trial169.stdout:527: oid 12 version 63 is already newer than 63 2026-02-25T08:44:16.226 INFO:tasks.rados.rados.0.trial169.stdout:527: oid 12 version 63 is already newer than 63 2026-02-25T08:44:16.226 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 12 v 63 (ObjNum 251 snap 72 seq_num 251) dirty exists 2026-02-25T08:44:16.226 INFO:tasks.rados.rados.0.trial169.stdout:527: left oid 12 (ObjNum 251 snap 72 seq_num 251) 2026-02-25T08:44:16.266 INFO:tasks.rados.rados.0.trial169.stdout:527: done (1 left) 2026-02-25T08:44:16.266 INFO:tasks.rados.rados.0.trial169.stdout:529: done (0 left) 2026-02-25T08:44:16.266 INFO:tasks.rados.rados.0.trial169.stdout:530: write (excl) oid 50 current snap is 72 2026-02-25T08:44:16.266 INFO:tasks.rados.rados.0.trial169.stdout:530: seq_num 253 ranges {652171=477646,1821709=430462} 2026-02-25T08:44:16.270 INFO:tasks.rados.rados.0.trial169.stdout:530: writing trial16941936-50 from 652171 to 1129817 tid 1 2026-02-25T08:44:16.274 INFO:tasks.rados.rados.0.trial169.stdout:530: writing trial16941936-50 from 1821709 to 2252171 tid 2 2026-02-25T08:44:16.275 INFO:tasks.rados.rados.0.trial169.stdout:531: snap_create 2026-02-25T08:44:16.297 INFO:tasks.rados.rados.0.trial169.stdout:530: finishing write tid 1 to trial16941936-50 2026-02-25T08:44:16.310 INFO:tasks.rados.rados.0.trial169.stdout:530: finishing write tid 2 to trial16941936-50 2026-02-25T08:44:16.311 INFO:tasks.rados.rados.0.trial169.stdout:530: finishing write tid 3 to trial16941936-50 2026-02-25T08:44:16.313 INFO:tasks.rados.rados.0.trial169.stdout:530: finishing write tid 4 to trial16941936-50 2026-02-25T08:44:16.313 INFO:tasks.rados.rados.0.trial169.stdout:530: oid 50 updating version 0 to 275 2026-02-25T08:44:16.313 INFO:tasks.rados.rados.0.trial169.stdout:530: oid 50 updating version 275 to 277 2026-02-25T08:44:16.313 INFO:tasks.rados.rados.0.trial169.stdout:530: oid 50 version 277 is already newer than 277 2026-02-25T08:44:16.313 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 50 v 277 (ObjNum 253 snap 72 seq_num 253) dirty exists 2026-02-25T08:44:16.313 INFO:tasks.rados.rados.0.trial169.stdout:530: left oid 50 (ObjNum 253 snap 72 seq_num 253) 2026-02-25T08:44:16.926 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:17.003 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:17.001+0000 7f19b6f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:17.006 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:17.004+0000 7f19b6f0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:17.010 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:17.049 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:17.050 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:17.230 INFO:tasks.rados.rados.0.trial169.stdout:530: done (1 left) 2026-02-25T08:44:17.230 INFO:tasks.rados.rados.0.trial169.stdout:531: done (0 left) 2026-02-25T08:44:17.230 INFO:tasks.rados.rados.0.trial169.stdout:532: read oid 17 snap -1 2026-02-25T08:44:17.230 INFO:tasks.rados.rados.0.trial169.stdout:532: expect deleted 2026-02-25T08:44:17.230 INFO:tasks.rados.rados.0.trial169.stdout:533: read oid 12 snap 66 2026-02-25T08:44:17.230 INFO:tasks.rados.rados.0.trial169.stdout:533: expect deleted 2026-02-25T08:44:17.230 INFO:tasks.rados.rados.0.trial169.stdout:534: read oid 35 snap -1 2026-02-25T08:44:17.230 INFO:tasks.rados.rados.0.trial169.stdout:534: expect deleted 2026-02-25T08:44:17.230 INFO:tasks.rados.rados.0.trial169.stdout:535: write (excl) oid 19 current snap is 73 2026-02-25T08:44:17.230 INFO:tasks.rados.rados.0.trial169.stdout:535: seq_num 254 ranges {668978=552895,1763838=505140} 2026-02-25T08:44:17.234 INFO:tasks.rados.rados.0.trial169.stdout:535: writing trial16941936-19 from 668978 to 1221873 tid 1 2026-02-25T08:44:17.238 INFO:tasks.rados.rados.0.trial169.stdout:535: writing trial16941936-19 from 1763838 to 2268978 tid 2 2026-02-25T08:44:17.238 INFO:tasks.rados.rados.0.trial169.stdout:536: snap_remove snap 27 2026-02-25T08:44:18.109 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~132s 2026-02-25T08:44:18.137 INFO:tasks.rados.rados.0.trial169.stdout:536: done (4 left) 2026-02-25T08:44:18.137 INFO:tasks.rados.rados.0.trial169.stdout:537: write (excl) oid 7 current snap is 73 2026-02-25T08:44:18.137 INFO:tasks.rados.rados.0.trial169.stdout:537: seq_num 255 ranges {685785=628144,1822320=463465} 2026-02-25T08:44:18.142 INFO:tasks.rados.rados.0.trial169.stdout:537: writing trial16941936-7 from 685785 to 1313929 tid 1 2026-02-25T08:44:18.146 INFO:tasks.rados.rados.0.trial169.stdout:537: writing trial16941936-7 from 1822320 to 2285785 tid 2 2026-02-25T08:44:18.146 INFO:tasks.rados.rados.0.trial169.stdout:538: read oid 24 snap -1 2026-02-25T08:44:18.146 INFO:tasks.rados.rados.0.trial169.stdout:538: expect (ObjNum 239 snap 68 seq_num 239) 2026-02-25T08:44:18.146 INFO:tasks.rados.rados.0.trial169.stdout:539: snap_create 2026-02-25T08:44:18.146 INFO:tasks.rados.rados.0.trial169.stdout:535: finishing write tid 1 to trial16941936-19 2026-02-25T08:44:18.146 INFO:tasks.rados.rados.0.trial169.stdout:535: finishing write tid 2 to trial16941936-19 2026-02-25T08:44:18.146 INFO:tasks.rados.rados.0.trial169.stdout:535: finishing write tid 3 to trial16941936-19 2026-02-25T08:44:18.147 INFO:tasks.rados.rados.0.trial169.stdout:535: finishing write tid 4 to trial16941936-19 2026-02-25T08:44:18.147 INFO:tasks.rados.rados.0.trial169.stdout:535: oid 19 updating version 0 to 69 2026-02-25T08:44:18.147 INFO:tasks.rados.rados.0.trial169.stdout:535: oid 19 version 69 is already newer than 67 2026-02-25T08:44:18.147 INFO:tasks.rados.rados.0.trial169.stdout:535: oid 19 version 69 is already newer than 69 2026-02-25T08:44:18.147 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 19 v 69 (ObjNum 254 snap 73 seq_num 254) dirty exists 2026-02-25T08:44:18.147 INFO:tasks.rados.rados.0.trial169.stdout:535: left oid 19 (ObjNum 254 snap 73 seq_num 254) 2026-02-25T08:44:18.176 INFO:tasks.rados.rados.0.trial169.stdout:537: finishing write tid 1 to trial16941936-7 2026-02-25T08:44:18.185 INFO:tasks.rados.rados.0.trial169.stdout:538: expect (ObjNum 239 snap 68 seq_num 239) 2026-02-25T08:44:18.209 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1735458 2026-02-25T08:44:18.334 INFO:tasks.rados.rados.0.trial169.stdout:537: finishing write tid 2 to trial16941936-7 2026-02-25T08:44:18.334 INFO:tasks.rados.rados.0.trial169.stdout:537: finishing write tid 3 to trial16941936-7 2026-02-25T08:44:18.334 INFO:tasks.rados.rados.0.trial169.stdout:537: finishing write tid 4 to trial16941936-7 2026-02-25T08:44:18.334 INFO:tasks.rados.rados.0.trial169.stdout:537: oid 7 updating version 0 to 282 2026-02-25T08:44:18.334 INFO:tasks.rados.rados.0.trial169.stdout:537: oid 7 updating version 282 to 284 2026-02-25T08:44:18.334 INFO:tasks.rados.rados.0.trial169.stdout:537: oid 7 version 284 is already newer than 284 2026-02-25T08:44:18.334 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 7 v 284 (ObjNum 255 snap 73 seq_num 255) dirty exists 2026-02-25T08:44:18.335 INFO:tasks.rados.rados.0.trial169.stdout:537: left oid 7 (ObjNum 255 snap 73 seq_num 255) 2026-02-25T08:44:19.051 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:19.130 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:19.127+0000 7f2af65b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:19.132 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:19.130+0000 7f2af65b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:19.137 INFO:tasks.rados.rados.0.trial169.stdout:532: done (6 left) 2026-02-25T08:44:19.137 INFO:tasks.rados.rados.0.trial169.stdout:533: done (5 left) 2026-02-25T08:44:19.137 INFO:tasks.rados.rados.0.trial169.stdout:534: done (4 left) 2026-02-25T08:44:19.137 INFO:tasks.rados.rados.0.trial169.stdout:535: done (3 left) 2026-02-25T08:44:19.137 INFO:tasks.rados.rados.0.trial169.stdout:537: done (2 left) 2026-02-25T08:44:19.137 INFO:tasks.rados.rados.0.trial169.stdout:538: done (1 left) 2026-02-25T08:44:19.137 INFO:tasks.rados.rados.0.trial169.stdout:539: done (0 left) 2026-02-25T08:44:19.137 INFO:tasks.rados.rados.0.trial169.stdout:540: snap_remove snap 71 2026-02-25T08:44:19.138 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:19.187 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:19.406 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:44:19.476 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:19.479 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:19.480 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:44:20.139 INFO:tasks.rados.rados.0.trial169.stdout:540: done (0 left) 2026-02-25T08:44:20.140 INFO:tasks.rados.rados.0.trial169.stdout:541: snap_remove snap 50 2026-02-25T08:44:21.145 INFO:tasks.rados.rados.0.trial169.stdout:541: done (0 left) 2026-02-25T08:44:21.145 INFO:tasks.rados.rados.0.trial169.stdout:542: write oid 26 current snap is 74 2026-02-25T08:44:21.145 INFO:tasks.rados.rados.0.trial169.stdout:542: seq_num 256 ranges {702592=703393,1880802=421790} 2026-02-25T08:44:21.151 INFO:tasks.rados.rados.0.trial169.stdout:542: writing trial16941936-26 from 702592 to 1405985 tid 1 2026-02-25T08:44:21.155 INFO:tasks.rados.rados.0.trial169.stdout:542: writing trial16941936-26 from 1880802 to 2302592 tid 2 2026-02-25T08:44:21.155 INFO:tasks.rados.rados.0.trial169.stdout:543: read oid 29 snap -1 2026-02-25T08:44:21.155 INFO:tasks.rados.rados.0.trial169.stdout:543: expect (ObjNum 231 snap 66 seq_num 231) 2026-02-25T08:44:21.155 INFO:tasks.rados.rados.0.trial169.stdout:544: write (excl) oid 41 current snap is 74 2026-02-25T08:44:21.155 INFO:tasks.rados.rados.0.trial169.stdout:544: seq_num 257 ranges {719399=778642,1939284=380115} 2026-02-25T08:44:21.160 INFO:tasks.rados.rados.0.trial169.stdout:544: writing trial16941936-41 from 719399 to 1498041 tid 1 2026-02-25T08:44:21.163 INFO:tasks.rados.rados.0.trial169.stdout:544: writing trial16941936-41 from 1939284 to 2319399 tid 2 2026-02-25T08:44:21.164 INFO:tasks.rados.rados.0.trial169.stdout:545: write oid 42 current snap is 74 2026-02-25T08:44:21.164 INFO:tasks.rados.rados.0.trial169.stdout:545: seq_num 258 ranges {736206=453891,1881413=454793} 2026-02-25T08:44:21.168 INFO:tasks.rados.rados.0.trial169.stdout:545: writing trial16941936-42 from 736206 to 1190097 tid 1 2026-02-25T08:44:21.172 INFO:tasks.rados.rados.0.trial169.stdout:545: writing trial16941936-42 from 1881413 to 2336206 tid 2 2026-02-25T08:44:21.172 INFO:tasks.rados.rados.0.trial169.stdout:546: write (excl) oid 46 current snap is 74 2026-02-25T08:44:21.172 INFO:tasks.rados.rados.0.trial169.stdout:546: seq_num 259 ranges {753013=645493,2056248=296765} 2026-02-25T08:44:21.177 INFO:tasks.rados.rados.0.trial169.stdout:546: writing trial16941936-46 from 753013 to 1398506 tid 1 2026-02-25T08:44:21.181 INFO:tasks.rados.rados.0.trial169.stdout:546: writing trial16941936-46 from 2056248 to 2353013 tid 2 2026-02-25T08:44:21.181 INFO:tasks.rados.rados.0.trial169.stdout:547: write (excl) oid 17 current snap is 74 2026-02-25T08:44:21.181 INFO:tasks.rados.rados.0.trial169.stdout:547: seq_num 260 ranges {769820=720742,2114730=255090} 2026-02-25T08:44:21.187 INFO:tasks.rados.rados.0.trial169.stdout:547: writing trial16941936-17 from 769820 to 1490562 tid 1 2026-02-25T08:44:21.188 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:21.191 INFO:tasks.rados.rados.0.trial169.stdout:547: writing trial16941936-17 from 2114730 to 2369820 tid 2 2026-02-25T08:44:21.191 INFO:tasks.rados.rados.0.trial169.stdout:548: write (excl) oid 10 current snap is 74 2026-02-25T08:44:21.191 INFO:tasks.rados.rados.0.trial169.stdout:548: seq_num 261 ranges {786627=795991,2173212=213415} 2026-02-25T08:44:21.196 INFO:tasks.rados.rados.0.trial169.stdout:548: writing trial16941936-10 from 786627 to 1582618 tid 1 2026-02-25T08:44:21.199 INFO:tasks.rados.rados.0.trial169.stdout:548: writing trial16941936-10 from 2173212 to 2386627 tid 2 2026-02-25T08:44:21.199 INFO:tasks.rados.rados.0.trial169.stdout:549: write (excl) oid 9 current snap is 74 2026-02-25T08:44:21.200 INFO:tasks.rados.rados.0.trial169.stdout:549: seq_num 262 ranges {403434=471240,1315341=400756} 2026-02-25T08:44:21.202 INFO:tasks.rados.rados.0.trial169.stdout:549: writing trial16941936-9 from 403434 to 874674 tid 1 2026-02-25T08:44:21.206 INFO:tasks.rados.rados.0.trial169.stdout:549: writing trial16941936-9 from 1315341 to 1716097 tid 2 2026-02-25T08:44:21.206 INFO:tasks.rados.rados.0.trial169.stdout:550: write (excl) oid 43 current snap is 74 2026-02-25T08:44:21.206 INFO:tasks.rados.rados.0.trial169.stdout:550: seq_num 263 ranges {420241=546489,1373823=544414} 2026-02-25T08:44:21.209 INFO:tasks.rados.rados.0.trial169.stdout:550: writing trial16941936-43 from 420241 to 966730 tid 1 2026-02-25T08:44:21.213 INFO:tasks.rados.rados.0.trial169.stdout:550: writing trial16941936-43 from 1373823 to 1918237 tid 2 2026-02-25T08:44:21.213 INFO:tasks.rados.rados.0.trial169.stdout:543: expect (ObjNum 231 snap 66 seq_num 231) 2026-02-25T08:44:21.263 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2616564 2026-02-25T08:44:21.268 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:21.265+0000 7f5e9b21a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:21.271 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:21.269+0000 7f5e9b21a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:21.274 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:21.318 INFO:tasks.rados.rados.0.trial169.stdout:545: finishing write tid 1 to trial16941936-42 2026-02-25T08:44:21.318 INFO:tasks.rados.rados.0.trial169.stdout:542: finishing write tid 1 to trial16941936-26 2026-02-25T08:44:21.318 INFO:tasks.rados.rados.0.trial169.stdout:545: finishing write tid 2 to trial16941936-42 2026-02-25T08:44:21.318 INFO:tasks.rados.rados.0.trial169.stdout:551: read oid 15 snap 68 2026-02-25T08:44:21.318 INFO:tasks.rados.rados.0.trial169.stdout:551: expect (ObjNum 191 snap 53 seq_num 191) 2026-02-25T08:44:21.318 INFO:tasks.rados.rados.0.trial169.stdout:543: done (9 left) 2026-02-25T08:44:21.318 INFO:tasks.rados.rados.0.trial169.stdout:552: read oid 20 snap -1 2026-02-25T08:44:21.318 INFO:tasks.rados.rados.0.trial169.stdout:552: expect (ObjNum 234 snap 67 seq_num 234) 2026-02-25T08:44:21.319 INFO:tasks.rados.rados.0.trial169.stdout:553: snap_remove snap 61 2026-02-25T08:44:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:21.346 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:22.149 INFO:tasks.rados.rados.0.trial169.stdout:553: done (10 left) 2026-02-25T08:44:22.150 INFO:tasks.rados.rados.0.trial169.stdout:554: write oid 3 current snap is 74 2026-02-25T08:44:22.150 INFO:tasks.rados.rados.0.trial169.stdout:547: finishing write tid 1 to trial16941936-17 2026-02-25T08:44:22.150 INFO:tasks.rados.rados.0.trial169.stdout:554: seq_num 264 ranges {437048=621738,1832305=404425} 2026-02-25T08:44:22.153 INFO:tasks.rados.rados.0.trial169.stdout:554: writing trial16941936-3 from 437048 to 1058786 tid 1 2026-02-25T08:44:22.160 INFO:tasks.rados.rados.0.trial169.stdout:554: writing trial16941936-3 from 1832305 to 2236730 tid 2 2026-02-25T08:44:22.160 INFO:tasks.rados.rados.0.trial169.stdout:555: read oid 49 snap -1 2026-02-25T08:44:22.161 INFO:tasks.rados.rados.0.trial169.stdout:555: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:44:22.161 INFO:tasks.rados.rados.0.trial169.stdout:556: delete oid 37 current snap is 74 2026-02-25T08:44:22.161 INFO:tasks.rados.rados.0.trial169.stdout:545: finishing write tid 3 to trial16941936-42 2026-02-25T08:44:22.161 INFO:tasks.rados.rados.0.trial169.stdout:542: finishing write tid 2 to trial16941936-26 2026-02-25T08:44:22.161 INFO:tasks.rados.rados.0.trial169.stdout:545: finishing write tid 4 to trial16941936-42 2026-02-25T08:44:22.161 INFO:tasks.rados.rados.0.trial169.stdout:545: oid 42 updating version 0 to 98 2026-02-25T08:44:22.161 INFO:tasks.rados.rados.0.trial169.stdout:545: oid 42 version 98 is already newer than 98 2026-02-25T08:44:22.161 INFO:tasks.rados.rados.0.trial169.stdout:545: oid 42 version 98 is already newer than 98 2026-02-25T08:44:22.161 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 42 v 98 (ObjNum 258 snap 74 seq_num 258) dirty exists 2026-02-25T08:44:22.161 INFO:tasks.rados.rados.0.trial169.stdout:545: left oid 42 (ObjNum 258 snap 74 seq_num 258) 2026-02-25T08:44:22.161 INFO:tasks.rados.rados.0.trial169.stdout:542: finishing write tid 3 to trial16941936-26 2026-02-25T08:44:22.162 INFO:tasks.rados.rados.0.trial169.stdout:542: finishing write tid 4 to trial16941936-26 2026-02-25T08:44:22.162 INFO:tasks.rados.rados.0.trial169.stdout:542: oid 26 updating version 0 to 220 2026-02-25T08:44:22.162 INFO:tasks.rados.rados.0.trial169.stdout:542: oid 26 version 220 is already newer than 220 2026-02-25T08:44:22.162 INFO:tasks.rados.rados.0.trial169.stdout:542: oid 26 version 220 is already newer than 220 2026-02-25T08:44:22.162 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 26 v 220 (ObjNum 256 snap 74 seq_num 256) dirty exists 2026-02-25T08:44:22.162 INFO:tasks.rados.rados.0.trial169.stdout:542: left oid 26 (ObjNum 256 snap 74 seq_num 256) 2026-02-25T08:44:22.162 INFO:tasks.rados.rados.0.trial169.stdout:547: finishing write tid 2 to trial16941936-17 2026-02-25T08:44:22.162 INFO:tasks.rados.rados.0.trial169.stdout:547: finishing write tid 3 to trial16941936-17 2026-02-25T08:44:22.162 INFO:tasks.rados.rados.0.trial169.stdout:549: finishing write tid 1 to trial16941936-9 2026-02-25T08:44:22.162 INFO:tasks.rados.rados.0.trial169.stdout:547: finishing write tid 4 to trial16941936-17 2026-02-25T08:44:22.162 INFO:tasks.rados.rados.0.trial169.stdout:547: oid 17 updating version 0 to 53 2026-02-25T08:44:22.162 INFO:tasks.rados.rados.0.trial169.stdout:547: oid 17 version 53 is already newer than 51 2026-02-25T08:44:22.163 INFO:tasks.rados.rados.0.trial169.stdout:547: oid 17 version 53 is already newer than 53 2026-02-25T08:44:22.163 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 17 v 53 (ObjNum 260 snap 74 seq_num 260) dirty exists 2026-02-25T08:44:22.163 INFO:tasks.rados.rados.0.trial169.stdout:547: left oid 17 (ObjNum 260 snap 74 seq_num 260) 2026-02-25T08:44:22.163 INFO:tasks.rados.rados.0.trial169.stdout:546: finishing write tid 1 to trial16941936-46 2026-02-25T08:44:22.163 INFO:tasks.rados.rados.0.trial169.stdout:550: finishing write tid 1 to trial16941936-43 2026-02-25T08:44:22.163 INFO:tasks.rados.rados.0.trial169.stdout:549: finishing write tid 2 to trial16941936-9 2026-02-25T08:44:22.163 INFO:tasks.rados.rados.0.trial169.stdout:549: finishing write tid 3 to trial16941936-9 2026-02-25T08:44:22.163 INFO:tasks.rados.rados.0.trial169.stdout:544: finishing write tid 1 to trial16941936-41 2026-02-25T08:44:22.163 INFO:tasks.rados.rados.0.trial169.stdout:550: finishing write tid 2 to trial16941936-43 2026-02-25T08:44:22.163 INFO:tasks.rados.rados.0.trial169.stdout:549: finishing write tid 4 to trial16941936-9 2026-02-25T08:44:22.163 INFO:tasks.rados.rados.0.trial169.stdout:549: oid 9 updating version 0 to 73 2026-02-25T08:44:22.163 INFO:tasks.rados.rados.0.trial169.stdout:549: oid 9 version 73 is already newer than 73 2026-02-25T08:44:22.163 INFO:tasks.rados.rados.0.trial169.stdout:549: oid 9 version 73 is already newer than 71 2026-02-25T08:44:22.164 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 9 v 73 (ObjNum 262 snap 74 seq_num 262) dirty exists 2026-02-25T08:44:22.164 INFO:tasks.rados.rados.0.trial169.stdout:549: left oid 9 (ObjNum 262 snap 74 seq_num 262) 2026-02-25T08:44:22.164 INFO:tasks.rados.rados.0.trial169.stdout:546: finishing write tid 2 to trial16941936-46 2026-02-25T08:44:22.164 INFO:tasks.rados.rados.0.trial169.stdout:550: finishing write tid 3 to trial16941936-43 2026-02-25T08:44:22.164 INFO:tasks.rados.rados.0.trial169.stdout:550: finishing write tid 4 to trial16941936-43 2026-02-25T08:44:22.164 INFO:tasks.rados.rados.0.trial169.stdout:550: oid 43 updating version 0 to 35 2026-02-25T08:44:22.164 INFO:tasks.rados.rados.0.trial169.stdout:550: oid 43 version 35 is already newer than 33 2026-02-25T08:44:22.164 INFO:tasks.rados.rados.0.trial169.stdout:550: oid 43 version 35 is already newer than 35 2026-02-25T08:44:22.164 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 43 v 35 (ObjNum 263 snap 74 seq_num 263) dirty exists 2026-02-25T08:44:22.164 INFO:tasks.rados.rados.0.trial169.stdout:550: left oid 43 (ObjNum 263 snap 74 seq_num 263) 2026-02-25T08:44:22.164 INFO:tasks.rados.rados.0.trial169.stdout:546: finishing write tid 3 to trial16941936-46 2026-02-25T08:44:22.164 INFO:tasks.rados.rados.0.trial169.stdout:546: finishing write tid 4 to trial16941936-46 2026-02-25T08:44:22.165 INFO:tasks.rados.rados.0.trial169.stdout:546: oid 46 updating version 0 to 102 2026-02-25T08:44:22.165 INFO:tasks.rados.rados.0.trial169.stdout:546: oid 46 version 102 is already newer than 100 2026-02-25T08:44:22.165 INFO:tasks.rados.rados.0.trial169.stdout:546: oid 46 version 102 is already newer than 102 2026-02-25T08:44:22.165 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 46 v 102 (ObjNum 259 snap 74 seq_num 259) dirty exists 2026-02-25T08:44:22.165 INFO:tasks.rados.rados.0.trial169.stdout:546: left oid 46 (ObjNum 259 snap 74 seq_num 259) 2026-02-25T08:44:22.165 INFO:tasks.rados.rados.0.trial169.stdout:544: finishing write tid 2 to trial16941936-41 2026-02-25T08:44:22.165 INFO:tasks.rados.rados.0.trial169.stdout:544: finishing write tid 3 to trial16941936-41 2026-02-25T08:44:22.165 INFO:tasks.rados.rados.0.trial169.stdout:544: finishing write tid 4 to trial16941936-41 2026-02-25T08:44:22.165 INFO:tasks.rados.rados.0.trial169.stdout:544: oid 41 updating version 0 to 224 2026-02-25T08:44:22.165 INFO:tasks.rados.rados.0.trial169.stdout:544: oid 41 version 224 is already newer than 222 2026-02-25T08:44:22.165 INFO:tasks.rados.rados.0.trial169.stdout:544: oid 41 version 224 is already newer than 224 2026-02-25T08:44:22.165 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 41 v 224 (ObjNum 257 snap 74 seq_num 257) dirty exists 2026-02-25T08:44:22.165 INFO:tasks.rados.rados.0.trial169.stdout:544: left oid 41 (ObjNum 257 snap 74 seq_num 257) 2026-02-25T08:44:22.165 INFO:tasks.rados.rados.0.trial169.stdout:548: finishing write tid 1 to trial16941936-10 2026-02-25T08:44:22.166 INFO:tasks.rados.rados.0.trial169.stdout:548: finishing write tid 2 to trial16941936-10 2026-02-25T08:44:22.166 INFO:tasks.rados.rados.0.trial169.stdout:548: finishing write tid 3 to trial16941936-10 2026-02-25T08:44:22.166 INFO:tasks.rados.rados.0.trial169.stdout:552: expect (ObjNum 234 snap 67 seq_num 234) 2026-02-25T08:44:22.258 INFO:tasks.rados.rados.0.trial169.stdout:548: finishing write tid 4 to trial16941936-10 2026-02-25T08:44:22.258 INFO:tasks.rados.rados.0.trial169.stdout:548: oid 10 updating version 0 to 66 2026-02-25T08:44:22.258 INFO:tasks.rados.rados.0.trial169.stdout:548: oid 10 version 66 is already newer than 66 2026-02-25T08:44:22.258 INFO:tasks.rados.rados.0.trial169.stdout:548: oid 10 version 66 is already newer than 64 2026-02-25T08:44:22.258 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 10 v 66 (ObjNum 261 snap 74 seq_num 261) dirty exists 2026-02-25T08:44:22.258 INFO:tasks.rados.rados.0.trial169.stdout:548: left oid 10 (ObjNum 261 snap 74 seq_num 261) 2026-02-25T08:44:22.258 INFO:tasks.rados.rados.0.trial169.stdout:551: expect (ObjNum 191 snap 53 seq_num 191) 2026-02-25T08:44:22.269 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 3122455 2026-02-25T08:44:22.270 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:22.323 INFO:tasks.rados.rados.0.trial169.stdout:555: expect (ObjNum 48 snap 0 seq_num 48) 2026-02-25T08:44:22.359 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:22.383 INFO:tasks.rados.rados.0.trial169.stdout:554: finishing write tid 1 to trial16941936-3 2026-02-25T08:44:22.383 INFO:tasks.rados.rados.0.trial169.stdout:554: finishing write tid 2 to trial16941936-3 2026-02-25T08:44:22.383 INFO:tasks.rados.rados.0.trial169.stdout:554: finishing write tid 3 to trial16941936-3 2026-02-25T08:44:22.383 INFO:tasks.rados.rados.0.trial169.stdout:554: finishing write tid 4 to trial16941936-3 2026-02-25T08:44:22.383 INFO:tasks.rados.rados.0.trial169.stdout:554: oid 3 updating version 0 to 105 2026-02-25T08:44:22.383 INFO:tasks.rados.rados.0.trial169.stdout:554: oid 3 version 105 is already newer than 105 2026-02-25T08:44:22.384 INFO:tasks.rados.rados.0.trial169.stdout:554: oid 3 version 105 is already newer than 105 2026-02-25T08:44:22.384 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 3 v 105 (ObjNum 264 snap 74 seq_num 264) dirty exists 2026-02-25T08:44:22.384 INFO:tasks.rados.rados.0.trial169.stdout:554: left oid 3 (ObjNum 264 snap 74 seq_num 264) 2026-02-25T08:44:22.384 INFO:tasks.rados.rados.0.trial169.stdout:542: done (12 left) 2026-02-25T08:44:22.384 INFO:tasks.rados.rados.0.trial169.stdout:544: done (11 left) 2026-02-25T08:44:22.384 INFO:tasks.rados.rados.0.trial169.stdout:545: done (10 left) 2026-02-25T08:44:22.384 INFO:tasks.rados.rados.0.trial169.stdout:546: done (9 left) 2026-02-25T08:44:22.384 INFO:tasks.rados.rados.0.trial169.stdout:547: done (8 left) 2026-02-25T08:44:22.384 INFO:tasks.rados.rados.0.trial169.stdout:548: done (7 left) 2026-02-25T08:44:22.384 INFO:tasks.rados.rados.0.trial169.stdout:549: done (6 left) 2026-02-25T08:44:22.384 INFO:tasks.rados.rados.0.trial169.stdout:550: done (5 left) 2026-02-25T08:44:22.384 INFO:tasks.rados.rados.0.trial169.stdout:551: done (4 left) 2026-02-25T08:44:22.385 INFO:tasks.rados.rados.0.trial169.stdout:552: done (3 left) 2026-02-25T08:44:22.385 INFO:tasks.rados.rados.0.trial169.stdout:554: done (2 left) 2026-02-25T08:44:22.385 INFO:tasks.rados.rados.0.trial169.stdout:555: done (1 left) 2026-02-25T08:44:22.385 INFO:tasks.rados.rados.0.trial169.stdout:556: done (0 left) 2026-02-25T08:44:22.385 INFO:tasks.rados.rados.0.trial169.stdout:557: write oid 9 current snap is 74 2026-02-25T08:44:22.385 INFO:tasks.rados.rados.0.trial169.stdout:557: seq_num 266 ranges {470662=772236,1832916=408094} 2026-02-25T08:44:22.387 INFO:tasks.rados.rados.0.trial169.stdout:557: writing trial16941936-9 from 470662 to 1242898 tid 1 2026-02-25T08:44:22.391 INFO:tasks.rados.rados.0.trial169.stdout:557: writing trial16941936-9 from 1832916 to 2241010 tid 2 2026-02-25T08:44:22.391 INFO:tasks.rados.rados.0.trial169.stdout:558: delete oid 30 current snap is 74 2026-02-25T08:44:22.412 INFO:tasks.rados.rados.0.trial169.stdout:557: finishing write tid 1 to trial16941936-9 2026-02-25T08:44:22.417 INFO:tasks.rados.rados.0.trial169.stdout:557: finishing write tid 2 to trial16941936-9 2026-02-25T08:44:22.421 INFO:tasks.rados.rados.0.trial169.stdout:557: finishing write tid 3 to trial16941936-9 2026-02-25T08:44:22.423 INFO:tasks.rados.rados.0.trial169.stdout:557: finishing write tid 4 to trial16941936-9 2026-02-25T08:44:22.423 INFO:tasks.rados.rados.0.trial169.stdout:557: oid 9 updating version 0 to 76 2026-02-25T08:44:22.423 INFO:tasks.rados.rados.0.trial169.stdout:557: oid 9 version 76 is already newer than 76 2026-02-25T08:44:22.423 INFO:tasks.rados.rados.0.trial169.stdout:557: oid 9 version 76 is already newer than 76 2026-02-25T08:44:22.423 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 9 v 76 (ObjNum 266 snap 74 seq_num 266) dirty exists 2026-02-25T08:44:22.423 INFO:tasks.rados.rados.0.trial169.stdout:557: left oid 9 (ObjNum 266 snap 74 seq_num 266) 2026-02-25T08:44:22.425 INFO:tasks.rados.rados.0.trial169.stdout:557: done (1 left) 2026-02-25T08:44:22.425 INFO:tasks.rados.rados.0.trial169.stdout:558: done (0 left) 2026-02-25T08:44:22.425 INFO:tasks.rados.rados.0.trial169.stdout:559: delete oid 9 current snap is 74 2026-02-25T08:44:22.443 INFO:tasks.rados.rados.0.trial169.stdout:559: done (0 left) 2026-02-25T08:44:22.443 INFO:tasks.rados.rados.0.trial169.stdout:560: delete oid 10 current snap is 74 2026-02-25T08:44:22.458 INFO:tasks.rados.rados.0.trial169.stdout:560: done (0 left) 2026-02-25T08:44:22.458 INFO:tasks.rados.rados.0.trial169.stdout:561: read oid 36 snap 69 2026-02-25T08:44:22.458 INFO:tasks.rados.rados.0.trial169.stdout:561: expect deleted 2026-02-25T08:44:22.458 INFO:tasks.rados.rados.0.trial169.stdout:562: snap_remove snap 57 2026-02-25T08:44:23.150 INFO:tasks.rados.rados.0.trial169.stdout:562: done (1 left) 2026-02-25T08:44:23.150 INFO:tasks.rados.rados.0.trial169.stdout:563: snap_remove snap 66 2026-02-25T08:44:23.347 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:23.426 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:23.423+0000 7ff9ad8b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:23.428 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:23.427+0000 7ff9ad8b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:23.432 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:23.472 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:23.473 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:23.613 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~138s 2026-02-25T08:44:24.154 INFO:tasks.rados.rados.0.trial169.stdout:563: done (1 left) 2026-02-25T08:44:24.154 INFO:tasks.rados.rados.0.trial169.stdout:564: snap_create 2026-02-25T08:44:24.481 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:44:24.554 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:24.557 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:44:25.159 INFO:tasks.rados.rados.0.trial169.stdout:561: done (1 left) 2026-02-25T08:44:25.159 INFO:tasks.rados.rados.0.trial169.stdout:564: done (0 left) 2026-02-25T08:44:25.159 INFO:tasks.rados.rados.0.trial169.stdout:565: snap_remove snap 68 2026-02-25T08:44:25.474 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:25.551 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:25.548+0000 7f4345035640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:25.553 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:25.551+0000 7f4345035640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:25.556 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:25.596 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:25.597 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:26.162 INFO:tasks.rados.rados.0.trial169.stdout:565: done (0 left) 2026-02-25T08:44:26.162 INFO:tasks.rados.rados.0.trial169.stdout:566: write oid 9 current snap is 75 2026-02-25T08:44:26.162 INFO:tasks.rados.rados.0.trial169.stdout:566: seq_num 270 ranges {537890=789585,1783197=699079} 2026-02-25T08:44:26.167 INFO:tasks.rados.rados.0.trial169.stdout:566: writing trial16941936-9 from 537890 to 1327475 tid 1 2026-02-25T08:44:26.172 INFO:tasks.rados.rados.0.trial169.stdout:566: writing trial16941936-9 from 1783197 to 2482276 tid 2 2026-02-25T08:44:26.172 INFO:tasks.rados.rados.0.trial169.stdout:567: read oid 33 snap -1 2026-02-25T08:44:26.172 INFO:tasks.rados.rados.0.trial169.stdout:567: expect (ObjNum 193 snap 54 seq_num 193) 2026-02-25T08:44:26.172 INFO:tasks.rados.rados.0.trial169.stdout:568: write oid 10 current snap is 75 2026-02-25T08:44:26.173 INFO:tasks.rados.rados.0.trial169.stdout:568: seq_num 271 ranges {554697=464834,1725326=559090} 2026-02-25T08:44:26.176 INFO:tasks.rados.rados.0.trial169.stdout:568: writing trial16941936-10 from 554697 to 1019531 tid 1 2026-02-25T08:44:26.181 INFO:tasks.rados.rados.0.trial169.stdout:568: writing trial16941936-10 from 1725326 to 2284416 tid 2 2026-02-25T08:44:26.181 INFO:tasks.rados.rados.0.trial169.stdout:569: delete oid 11 current snap is 75 2026-02-25T08:44:26.202 INFO:tasks.rados.rados.0.trial169.stdout:566: finishing write tid 1 to trial16941936-9 2026-02-25T08:44:26.207 INFO:tasks.rados.rados.0.trial169.stdout:568: finishing write tid 1 to trial16941936-10 2026-02-25T08:44:26.211 INFO:tasks.rados.rados.0.trial169.stdout:566: finishing write tid 2 to trial16941936-9 2026-02-25T08:44:26.212 INFO:tasks.rados.rados.0.trial169.stdout:568: finishing write tid 2 to trial16941936-10 2026-02-25T08:44:26.216 INFO:tasks.rados.rados.0.trial169.stdout:568: finishing write tid 3 to trial16941936-10 2026-02-25T08:44:26.218 INFO:tasks.rados.rados.0.trial169.stdout:568: finishing write tid 4 to trial16941936-10 2026-02-25T08:44:26.218 INFO:tasks.rados.rados.0.trial169.stdout:568: oid 10 updating version 0 to 74 2026-02-25T08:44:26.218 INFO:tasks.rados.rados.0.trial169.stdout:568: oid 10 version 74 is already newer than 74 2026-02-25T08:44:26.218 INFO:tasks.rados.rados.0.trial169.stdout:568: oid 10 version 74 is already newer than 74 2026-02-25T08:44:26.218 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 10 v 74 (ObjNum 271 snap 75 seq_num 271) dirty exists 2026-02-25T08:44:26.218 INFO:tasks.rados.rados.0.trial169.stdout:568: left oid 10 (ObjNum 271 snap 75 seq_num 271) 2026-02-25T08:44:26.219 INFO:tasks.rados.rados.0.trial169.stdout:566: finishing write tid 3 to trial16941936-9 2026-02-25T08:44:26.221 INFO:tasks.rados.rados.0.trial169.stdout:566: finishing write tid 4 to trial16941936-9 2026-02-25T08:44:26.221 INFO:tasks.rados.rados.0.trial169.stdout:566: oid 9 updating version 0 to 80 2026-02-25T08:44:26.221 INFO:tasks.rados.rados.0.trial169.stdout:566: oid 9 version 80 is already newer than 80 2026-02-25T08:44:26.222 INFO:tasks.rados.rados.0.trial169.stdout:566: oid 9 version 80 is already newer than 80 2026-02-25T08:44:26.222 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 9 v 80 (ObjNum 270 snap 75 seq_num 270) dirty exists 2026-02-25T08:44:26.222 INFO:tasks.rados.rados.0.trial169.stdout:566: left oid 9 (ObjNum 270 snap 75 seq_num 270) 2026-02-25T08:44:26.226 INFO:tasks.rados.rados.0.trial169.stdout:567: expect (ObjNum 193 snap 54 seq_num 193) 2026-02-25T08:44:26.325 INFO:tasks.rados.rados.0.trial169.stdout:566: done (3 left) 2026-02-25T08:44:26.325 INFO:tasks.rados.rados.0.trial169.stdout:567: done (2 left) 2026-02-25T08:44:26.325 INFO:tasks.rados.rados.0.trial169.stdout:568: done (1 left) 2026-02-25T08:44:26.325 INFO:tasks.rados.rados.0.trial169.stdout:569: done (0 left) 2026-02-25T08:44:26.325 INFO:tasks.rados.rados.0.trial169.stdout:570: snap_remove snap 70 2026-02-25T08:44:27.167 INFO:tasks.rados.rados.0.trial169.stdout:570: done (0 left) 2026-02-25T08:44:27.167 INFO:tasks.rados.rados.0.trial169.stdout:571: read oid 25 snap -1 2026-02-25T08:44:27.168 INFO:tasks.rados.rados.0.trial169.stdout:571: expect (ObjNum 245 snap 69 seq_num 245) 2026-02-25T08:44:27.168 INFO:tasks.rados.rados.0.trial169.stdout:572: delete oid 19 current snap is 75 2026-02-25T08:44:27.216 INFO:tasks.rados.rados.0.trial169.stdout:572: done (1 left) 2026-02-25T08:44:27.216 INFO:tasks.rados.rados.0.trial169.stdout:573: delete oid 10 current snap is 75 2026-02-25T08:44:27.227 INFO:tasks.rados.rados.0.trial169.stdout:571: expect (ObjNum 245 snap 69 seq_num 245) 2026-02-25T08:44:27.277 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1931878 2026-02-25T08:44:27.306 INFO:tasks.rados.rados.0.trial169.stdout:571: done (1 left) 2026-02-25T08:44:27.307 INFO:tasks.rados.rados.0.trial169.stdout:573: done (0 left) 2026-02-25T08:44:27.307 INFO:tasks.rados.rados.0.trial169.stdout:574: read oid 44 snap -1 2026-02-25T08:44:27.307 INFO:tasks.rados.rados.0.trial169.stdout:574: expect (ObjNum 220 snap 64 seq_num 220) 2026-02-25T08:44:27.307 INFO:tasks.rados.rados.0.trial169.stdout:575: snap_create 2026-02-25T08:44:27.319 INFO:tasks.rados.rados.0.trial169.stdout:574: expect (ObjNum 220 snap 64 seq_num 220) 2026-02-25T08:44:27.386 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:27.597 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:27.671 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:27.669+0000 7f9340b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:27.674 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:27.672+0000 7f9340b0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:27.678 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:27.716 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:28.140 INFO:tasks.rados.rados.0.trial169.stdout:574: done (1 left) 2026-02-25T08:44:28.140 INFO:tasks.rados.rados.0.trial169.stdout:575: done (0 left) 2026-02-25T08:44:28.140 INFO:tasks.rados.rados.0.trial169.stdout:576: delete oid 37 current snap is 76 2026-02-25T08:44:28.155 INFO:tasks.rados.rados.0.trial169.stdout:576: done (0 left) 2026-02-25T08:44:28.155 INFO:tasks.rados.rados.0.trial169.stdout:577: read oid 47 snap -1 2026-02-25T08:44:28.155 INFO:tasks.rados.rados.0.trial169.stdout:577: expect (ObjNum 246 snap 69 seq_num 246) 2026-02-25T08:44:28.155 INFO:tasks.rados.rados.0.trial169.stdout:578: delete oid 6 current snap is 76 2026-02-25T08:44:28.170 INFO:tasks.rados.rados.0.trial169.stdout:577: expect (ObjNum 246 snap 69 seq_num 246) 2026-02-25T08:44:28.189 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:28.225 INFO:tasks.rados.rados.0.trial169.stdout:577: done (1 left) 2026-02-25T08:44:28.225 INFO:tasks.rados.rados.0.trial169.stdout:578: done (0 left) 2026-02-25T08:44:28.225 INFO:tasks.rados.rados.0.trial169.stdout:579: snap_remove snap 45 2026-02-25T08:44:29.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~143s 2026-02-25T08:44:29.172 INFO:tasks.rados.rados.0.trial169.stdout:579: done (0 left) 2026-02-25T08:44:29.173 INFO:tasks.rados.rados.0.trial169.stdout:580: read oid 44 snap -1 2026-02-25T08:44:29.173 INFO:tasks.rados.rados.0.trial169.stdout:580: expect (ObjNum 220 snap 64 seq_num 220) 2026-02-25T08:44:29.173 INFO:tasks.rados.rados.0.trial169.stdout:581: snap_remove snap 75 2026-02-25T08:44:29.558 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:44:29.629 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:29.632 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:29.633 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:44:29.717 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:29.796 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:29.793+0000 7fbff365a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:29.798 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:29.796+0000 7fbff365a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:29.801 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:29.842 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:29.843 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:30.174 INFO:tasks.rados.rados.0.trial169.stdout:581: done (1 left) 2026-02-25T08:44:30.174 INFO:tasks.rados.rados.0.trial169.stdout:582: write (excl) oid 43 current snap is 76 2026-02-25T08:44:30.174 INFO:tasks.rados.rados.0.trial169.stdout:582: seq_num 277 ranges {655539=632681,2076218=570097} 2026-02-25T08:44:30.178 INFO:tasks.rados.rados.0.trial169.stdout:582: writing trial16941936-43 from 655539 to 1288220 tid 1 2026-02-25T08:44:30.184 INFO:tasks.rados.rados.0.trial169.stdout:582: writing trial16941936-43 from 2076218 to 2646315 tid 2 2026-02-25T08:44:30.184 INFO:tasks.rados.rados.0.trial169.stdout:583: write oid 4 current snap is 76 2026-02-25T08:44:30.184 INFO:tasks.rados.rados.0.trial169.stdout:583: seq_num 278 ranges {672346=707930,2134700=537646} 2026-02-25T08:44:30.189 INFO:tasks.rados.rados.0.trial169.stdout:583: writing trial16941936-4 from 672346 to 1380276 tid 1 2026-02-25T08:44:30.194 INFO:tasks.rados.rados.0.trial169.stdout:583: writing trial16941936-4 from 2134700 to 2672346 tid 2 2026-02-25T08:44:30.194 INFO:tasks.rados.rados.0.trial169.stdout:584: read oid 29 snap -1 2026-02-25T08:44:30.194 INFO:tasks.rados.rados.0.trial169.stdout:584: expect (ObjNum 231 snap 66 seq_num 231) 2026-02-25T08:44:30.195 INFO:tasks.rados.rados.0.trial169.stdout:585: delete oid 33 current snap is 76 2026-02-25T08:44:30.195 INFO:tasks.rados.rados.0.trial169.stdout:580: expect (ObjNum 220 snap 64 seq_num 220) 2026-02-25T08:44:30.281 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:30.324 INFO:tasks.rados.rados.0.trial169.stdout:580: done (4 left) 2026-02-25T08:44:30.325 INFO:tasks.rados.rados.0.trial169.stdout:585: done (3 left) 2026-02-25T08:44:30.325 INFO:tasks.rados.rados.0.trial169.stdout:586: read oid 10 snap -1 2026-02-25T08:44:30.325 INFO:tasks.rados.rados.0.trial169.stdout:586: expect deleted 2026-02-25T08:44:30.325 INFO:tasks.rados.rados.0.trial169.stdout:587: write oid 49 current snap is 76 2026-02-25T08:44:30.325 INFO:tasks.rados.rados.0.trial169.stdout:583: finishing write tid 1 to trial16941936-4 2026-02-25T08:44:30.325 INFO:tasks.rados.rados.0.trial169.stdout:587: seq_num 280 ranges {705960=458428,1735311=717424} 2026-02-25T08:44:30.329 INFO:tasks.rados.rados.0.trial169.stdout:587: writing trial16941936-49 from 705960 to 1164388 tid 1 2026-02-25T08:44:30.335 INFO:tasks.rados.rados.0.trial169.stdout:587: writing trial16941936-49 from 1735311 to 2452735 tid 2 2026-02-25T08:44:30.335 INFO:tasks.rados.rados.0.trial169.stdout:588: delete oid 42 current snap is 76 2026-02-25T08:44:30.335 INFO:tasks.rados.rados.0.trial169.stdout:582: finishing write tid 1 to trial16941936-43 2026-02-25T08:44:30.335 INFO:tasks.rados.rados.0.trial169.stdout:583: finishing write tid 2 to trial16941936-4 2026-02-25T08:44:30.335 INFO:tasks.rados.rados.0.trial169.stdout:582: finishing write tid 2 to trial16941936-43 2026-02-25T08:44:30.335 INFO:tasks.rados.rados.0.trial169.stdout:582: finishing write tid 3 to trial16941936-43 2026-02-25T08:44:30.335 INFO:tasks.rados.rados.0.trial169.stdout:583: finishing write tid 3 to trial16941936-4 2026-02-25T08:44:30.335 INFO:tasks.rados.rados.0.trial169.stdout:582: finishing write tid 4 to trial16941936-43 2026-02-25T08:44:30.336 INFO:tasks.rados.rados.0.trial169.stdout:582: oid 43 updating version 0 to 39 2026-02-25T08:44:30.336 INFO:tasks.rados.rados.0.trial169.stdout:582: oid 43 updating version 39 to 41 2026-02-25T08:44:30.336 INFO:tasks.rados.rados.0.trial169.stdout:582: oid 43 version 41 is already newer than 41 2026-02-25T08:44:30.336 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 43 v 41 (ObjNum 277 snap 76 seq_num 277) dirty exists 2026-02-25T08:44:30.336 INFO:tasks.rados.rados.0.trial169.stdout:582: left oid 43 (ObjNum 277 snap 76 seq_num 277) 2026-02-25T08:44:30.336 INFO:tasks.rados.rados.0.trial169.stdout:583: finishing write tid 4 to trial16941936-4 2026-02-25T08:44:30.336 INFO:tasks.rados.rados.0.trial169.stdout:583: oid 4 updating version 0 to 309 2026-02-25T08:44:30.336 INFO:tasks.rados.rados.0.trial169.stdout:583: oid 4 version 309 is already newer than 309 2026-02-25T08:44:30.336 INFO:tasks.rados.rados.0.trial169.stdout:583: oid 4 version 309 is already newer than 309 2026-02-25T08:44:30.336 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 4 v 309 (ObjNum 278 snap 76 seq_num 278) dirty exists 2026-02-25T08:44:30.336 INFO:tasks.rados.rados.0.trial169.stdout:583: left oid 4 (ObjNum 278 snap 76 seq_num 278) 2026-02-25T08:44:30.336 INFO:tasks.rados.rados.0.trial169.stdout:584: expect (ObjNum 231 snap 66 seq_num 231) 2026-02-25T08:44:30.439 INFO:tasks.rados.rados.0.trial169.stdout:582: done (5 left) 2026-02-25T08:44:30.440 INFO:tasks.rados.rados.0.trial169.stdout:583: done (4 left) 2026-02-25T08:44:30.440 INFO:tasks.rados.rados.0.trial169.stdout:584: done (3 left) 2026-02-25T08:44:30.440 INFO:tasks.rados.rados.0.trial169.stdout:588: done (2 left) 2026-02-25T08:44:30.440 INFO:tasks.rados.rados.0.trial169.stdout:589: write (excl) oid 22 current snap is 76 2026-02-25T08:44:30.440 INFO:tasks.rados.rados.0.trial169.stdout:587: finishing write tid 1 to trial16941936-49 2026-02-25T08:44:30.440 INFO:tasks.rados.rados.0.trial169.stdout:587: finishing write tid 2 to trial16941936-49 2026-02-25T08:44:30.440 INFO:tasks.rados.rados.0.trial169.stdout:587: finishing write tid 3 to trial16941936-49 2026-02-25T08:44:30.440 INFO:tasks.rados.rados.0.trial169.stdout:587: finishing write tid 4 to trial16941936-49 2026-02-25T08:44:30.440 INFO:tasks.rados.rados.0.trial169.stdout:587: oid 49 updating version 0 to 242 2026-02-25T08:44:30.441 INFO:tasks.rados.rados.0.trial169.stdout:587: oid 49 version 242 is already newer than 242 2026-02-25T08:44:30.441 INFO:tasks.rados.rados.0.trial169.stdout:587: oid 49 version 242 is already newer than 242 2026-02-25T08:44:30.441 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 49 v 242 (ObjNum 280 snap 76 seq_num 280) dirty exists 2026-02-25T08:44:30.441 INFO:tasks.rados.rados.0.trial169.stdout:587: left oid 49 (ObjNum 280 snap 76 seq_num 280) 2026-02-25T08:44:30.441 INFO:tasks.rados.rados.0.trial169.stdout:589: seq_num 282 ranges {739574=725279,1968628=721093} 2026-02-25T08:44:30.445 INFO:tasks.rados.rados.0.trial169.stdout:589: writing trial16941936-22 from 739574 to 1464853 tid 1 2026-02-25T08:44:30.450 INFO:tasks.rados.rados.0.trial169.stdout:589: writing trial16941936-22 from 1968628 to 2689721 tid 2 2026-02-25T08:44:30.450 INFO:tasks.rados.rados.0.trial169.stdout:586: done (2 left) 2026-02-25T08:44:30.450 INFO:tasks.rados.rados.0.trial169.stdout:587: done (1 left) 2026-02-25T08:44:30.451 INFO:tasks.rados.rados.0.trial169.stdout:590: snap_create 2026-02-25T08:44:30.460 INFO:tasks.rados.rados.0.trial169.stdout:589: finishing write tid 1 to trial16941936-22 2026-02-25T08:44:30.472 INFO:tasks.rados.rados.0.trial169.stdout:589: finishing write tid 2 to trial16941936-22 2026-02-25T08:44:30.474 INFO:tasks.rados.rados.0.trial169.stdout:589: finishing write tid 3 to trial16941936-22 2026-02-25T08:44:30.476 INFO:tasks.rados.rados.0.trial169.stdout:589: finishing write tid 4 to trial16941936-22 2026-02-25T08:44:30.476 INFO:tasks.rados.rados.0.trial169.stdout:589: oid 22 updating version 0 to 129 2026-02-25T08:44:30.476 INFO:tasks.rados.rados.0.trial169.stdout:589: oid 22 version 129 is already newer than 129 2026-02-25T08:44:30.476 INFO:tasks.rados.rados.0.trial169.stdout:589: oid 22 version 129 is already newer than 127 2026-02-25T08:44:30.476 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 22 v 129 (ObjNum 282 snap 76 seq_num 282) dirty exists 2026-02-25T08:44:30.476 INFO:tasks.rados.rados.0.trial169.stdout:589: left oid 22 (ObjNum 282 snap 76 seq_num 282) 2026-02-25T08:44:31.177 INFO:tasks.rados.rados.0.trial169.stdout:589: done (1 left) 2026-02-25T08:44:31.177 INFO:tasks.rados.rados.0.trial169.stdout:590: done (0 left) 2026-02-25T08:44:31.178 INFO:tasks.rados.rados.0.trial169.stdout:591: delete oid 23 current snap is 77 2026-02-25T08:44:31.191 INFO:tasks.rados.rados.0.trial169.stdout:591: done (0 left) 2026-02-25T08:44:31.191 INFO:tasks.rados.rados.0.trial169.stdout:592: delete oid 27 current snap is 77 2026-02-25T08:44:31.216 INFO:tasks.rados.rados.0.trial169.stdout:592: done (0 left) 2026-02-25T08:44:31.216 INFO:tasks.rados.rados.0.trial169.stdout:593: delete oid 18 current snap is 77 2026-02-25T08:44:31.240 INFO:tasks.rados.rados.0.trial169.stdout:593: done (0 left) 2026-02-25T08:44:31.240 INFO:tasks.rados.rados.0.trial169.stdout:594: delete oid 39 current snap is 77 2026-02-25T08:44:31.264 INFO:tasks.rados.rados.0.trial169.stdout:594: done (0 left) 2026-02-25T08:44:31.265 INFO:tasks.rados.rados.0.trial169.stdout:595: snap_remove snap 73 2026-02-25T08:44:31.844 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:31.922 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:31.919+0000 7f2b4c5d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:31.924 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:31.922+0000 7f2b4c5d3640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:31.927 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:31.969 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:32.183 INFO:tasks.rados.rados.0.trial169.stdout:595: done (0 left) 2026-02-25T08:44:32.183 INFO:tasks.rados.rados.0.trial169.stdout:596: read oid 6 snap -1 2026-02-25T08:44:32.183 INFO:tasks.rados.rados.0.trial169.stdout:596: expect deleted 2026-02-25T08:44:32.184 INFO:tasks.rados.rados.0.trial169.stdout:597: read oid 35 snap -1 2026-02-25T08:44:32.184 INFO:tasks.rados.rados.0.trial169.stdout:597: expect deleted 2026-02-25T08:44:32.184 INFO:tasks.rados.rados.0.trial169.stdout:598: snap_remove snap 72 2026-02-25T08:44:33.185 INFO:tasks.rados.rados.0.trial169.stdout:598: done (2 left) 2026-02-25T08:44:33.185 INFO:tasks.rados.rados.0.trial169.stdout:599: write (excl) oid 38 current snap is 77 2026-02-25T08:44:33.185 INFO:tasks.rados.rados.0.trial169.stdout:599: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2026-02-25T08:44:33.189 INFO:tasks.rados.rados.0.trial169.stdout:599: writing trial16941936-38 from 423609 to 1125133 tid 1 2026-02-25T08:44:33.193 INFO:tasks.rados.rados.0.trial169.stdout:599: writing trial16941936-38 from 1628332 to 2100421 tid 2 2026-02-25T08:44:33.196 INFO:tasks.rados.rados.0.trial169.stdout:599: writing trial16941936-38 from 2648987 to 2823609 tid 3 2026-02-25T08:44:33.196 INFO:tasks.rados.rados.0.trial169.stdout:600: snap_remove snap 69 2026-02-25T08:44:33.971 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:34.048 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:34.045+0000 7fded6acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:34.050 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:34.049+0000 7fded6acb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:34.054 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:34.094 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:34.094 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:34.191 INFO:tasks.rados.rados.0.trial169.stdout:600: done (3 left) 2026-02-25T08:44:34.191 INFO:tasks.rados.rados.0.trial169.stdout:601: snap_create 2026-02-25T08:44:34.192 INFO:tasks.rados.rados.0.trial169.stdout:599: finishing write tid 1 to trial16941936-38 2026-02-25T08:44:34.192 INFO:tasks.rados.rados.0.trial169.stdout:599: finishing write tid 2 to trial16941936-38 2026-02-25T08:44:34.192 INFO:tasks.rados.rados.0.trial169.stdout:599: finishing write tid 3 to trial16941936-38 2026-02-25T08:44:34.192 INFO:tasks.rados.rados.0.trial169.stdout:599: finishing write tid 4 to trial16941936-38 2026-02-25T08:44:34.192 INFO:tasks.rados.rados.0.trial169.stdout:599: finishing write tid 5 to trial16941936-38 2026-02-25T08:44:34.192 INFO:tasks.rados.rados.0.trial169.stdout:599: oid 38 updating version 0 to 53 2026-02-25T08:44:34.192 INFO:tasks.rados.rados.0.trial169.stdout:599: oid 38 version 53 is already newer than 50 2026-02-25T08:44:34.192 INFO:tasks.rados.rados.0.trial169.stdout:599: oid 38 version 53 is already newer than 53 2026-02-25T08:44:34.192 INFO:tasks.rados.rados.0.trial169.stdout:599: oid 38 version 53 is already newer than 53 2026-02-25T08:44:34.192 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 38 v 53 (ObjNum 287 snap 77 seq_num 287) dirty exists 2026-02-25T08:44:34.192 INFO:tasks.rados.rados.0.trial169.stdout:599: left oid 38 (ObjNum 287 snap 77 seq_num 287) 2026-02-25T08:44:34.621 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~149s 2026-02-25T08:44:34.634 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:44:34.707 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:34.710 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:34.711 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:44:35.194 INFO:tasks.rados.rados.0.trial169.stdout:596: done (3 left) 2026-02-25T08:44:35.195 INFO:tasks.rados.rados.0.trial169.stdout:597: done (2 left) 2026-02-25T08:44:35.195 INFO:tasks.rados.rados.0.trial169.stdout:599: done (1 left) 2026-02-25T08:44:35.195 INFO:tasks.rados.rados.0.trial169.stdout:601: done (0 left) 2026-02-25T08:44:35.195 INFO:tasks.rados.rados.0.trial169.stdout:602: snap_create 2026-02-25T08:44:36.095 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:36.171 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:36.168+0000 7fd23cc46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:36.174 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:36.172+0000 7fd23cc46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:36.177 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:36.196 INFO:tasks.rados.rados.0.trial169.stdout:602: done (0 left) 2026-02-25T08:44:36.196 INFO:tasks.rados.rados.0.trial169.stdout:603: read oid 1 snap -1 2026-02-25T08:44:36.196 INFO:tasks.rados.rados.0.trial169.stdout:603: expect deleted 2026-02-25T08:44:36.196 INFO:tasks.rados.rados.0.trial169.stdout:604: write oid 6 current snap is 79 2026-02-25T08:44:36.196 INFO:tasks.rados.rados.0.trial169.stdout:604: seq_num 288 ranges {440416=776773,1686814=732100} 2026-02-25T08:44:36.200 INFO:tasks.rados.rados.0.trial169.stdout:604: writing trial16941936-6 from 440416 to 1217189 tid 1 2026-02-25T08:44:36.206 INFO:tasks.rados.rados.0.trial169.stdout:604: writing trial16941936-6 from 1686814 to 2418914 tid 2 2026-02-25T08:44:36.206 INFO:tasks.rados.rados.0.trial169.stdout:605: read oid 41 snap -1 2026-02-25T08:44:36.206 INFO:tasks.rados.rados.0.trial169.stdout:605: expect (ObjNum 257 snap 74 seq_num 257) 2026-02-25T08:44:36.206 INFO:tasks.rados.rados.0.trial169.stdout:606: read oid 2 snap -1 2026-02-25T08:44:36.206 INFO:tasks.rados.rados.0.trial169.stdout:606: expect deleted 2026-02-25T08:44:36.206 INFO:tasks.rados.rados.0.trial169.stdout:607: write (excl) oid 47 current snap is 79 2026-02-25T08:44:36.206 INFO:tasks.rados.rados.0.trial169.stdout:607: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2026-02-25T08:44:36.209 INFO:tasks.rados.rados.0.trial169.stdout:607: writing trial16941936-47 from 457223 to 1025598 tid 1 2026-02-25T08:44:36.213 INFO:tasks.rados.rados.0.trial169.stdout:607: writing trial16941936-47 from 1461649 to 1937407 tid 2 2026-02-25T08:44:36.216 INFO:tasks.rados.rados.0.trial169.stdout:607: writing trial16941936-47 from 2420186 to 2857223 tid 3 2026-02-25T08:44:36.217 INFO:tasks.rados.rados.0.trial169.stdout:608: read oid 19 snap 78 2026-02-25T08:44:36.217 INFO:tasks.rados.rados.0.trial169.stdout:608: expect deleted 2026-02-25T08:44:36.217 INFO:tasks.rados.rados.0.trial169.stdout:609: write (excl) oid 49 current snap is 79 2026-02-25T08:44:36.217 INFO:tasks.rados.rados.0.trial169.stdout:609: seq_num 290 ranges {474030=643624,1520131=735769} 2026-02-25T08:44:36.220 INFO:tasks.rados.rados.0.trial169.stdout:609: writing trial16941936-49 from 474030 to 1117654 tid 1 2026-02-25T08:44:36.225 INFO:tasks.rados.rados.0.trial169.stdout:609: writing trial16941936-49 from 1520131 to 2255900 tid 2 2026-02-25T08:44:36.225 INFO:tasks.rados.rados.0.trial169.stdout:610: snap_create 2026-02-25T08:44:36.228 INFO:tasks.rados.rados.0.trial169.stdout:604: finishing write tid 1 to trial16941936-6 2026-02-25T08:44:36.234 INFO:tasks.rados.rados.0.trial169.stdout:604: finishing write tid 2 to trial16941936-6 2026-02-25T08:44:36.239 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:36.240 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:36.241 INFO:tasks.rados.rados.0.trial169.stdout:604: finishing write tid 3 to trial16941936-6 2026-02-25T08:44:36.243 INFO:tasks.rados.rados.0.trial169.stdout:604: finishing write tid 4 to trial16941936-6 2026-02-25T08:44:36.243 INFO:tasks.rados.rados.0.trial169.stdout:604: oid 6 updating version 0 to 46 2026-02-25T08:44:36.243 INFO:tasks.rados.rados.0.trial169.stdout:604: oid 6 version 46 is already newer than 46 2026-02-25T08:44:36.243 INFO:tasks.rados.rados.0.trial169.stdout:604: oid 6 version 46 is already newer than 46 2026-02-25T08:44:36.243 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 6 v 46 (ObjNum 288 snap 79 seq_num 288) dirty exists 2026-02-25T08:44:36.243 INFO:tasks.rados.rados.0.trial169.stdout:604: left oid 6 (ObjNum 288 snap 79 seq_num 288) 2026-02-25T08:44:36.245 INFO:tasks.rados.rados.0.trial169.stdout:605: expect (ObjNum 257 snap 74 seq_num 257) 2026-02-25T08:44:36.298 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 606465 2026-02-25T08:44:36.322 INFO:tasks.rados.rados.0.trial169.stdout:607: finishing write tid 1 to trial16941936-47 2026-02-25T08:44:36.322 INFO:tasks.rados.rados.0.trial169.stdout:609: finishing write tid 1 to trial16941936-49 2026-02-25T08:44:36.322 INFO:tasks.rados.rados.0.trial169.stdout:609: finishing write tid 2 to trial16941936-49 2026-02-25T08:44:36.322 INFO:tasks.rados.rados.0.trial169.stdout:607: finishing write tid 2 to trial16941936-47 2026-02-25T08:44:36.322 INFO:tasks.rados.rados.0.trial169.stdout:609: finishing write tid 3 to trial16941936-49 2026-02-25T08:44:36.322 INFO:tasks.rados.rados.0.trial169.stdout:609: finishing write tid 4 to trial16941936-49 2026-02-25T08:44:36.322 INFO:tasks.rados.rados.0.trial169.stdout:609: oid 49 updating version 0 to 256 2026-02-25T08:44:36.322 INFO:tasks.rados.rados.0.trial169.stdout:609: oid 49 updating version 256 to 258 2026-02-25T08:44:36.323 INFO:tasks.rados.rados.0.trial169.stdout:609: oid 49 version 258 is already newer than 258 2026-02-25T08:44:36.323 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 49 v 258 (ObjNum 290 snap 79 seq_num 290) dirty exists 2026-02-25T08:44:36.323 INFO:tasks.rados.rados.0.trial169.stdout:609: left oid 49 (ObjNum 290 snap 79 seq_num 290) 2026-02-25T08:44:36.323 INFO:tasks.rados.rados.0.trial169.stdout:607: finishing write tid 3 to trial16941936-47 2026-02-25T08:44:36.323 INFO:tasks.rados.rados.0.trial169.stdout:607: finishing write tid 4 to trial16941936-47 2026-02-25T08:44:36.323 INFO:tasks.rados.rados.0.trial169.stdout:607: finishing write tid 5 to trial16941936-47 2026-02-25T08:44:36.323 INFO:tasks.rados.rados.0.trial169.stdout:607: oid 47 updating version 0 to 58 2026-02-25T08:44:36.323 INFO:tasks.rados.rados.0.trial169.stdout:607: oid 47 version 58 is already newer than 55 2026-02-25T08:44:36.323 INFO:tasks.rados.rados.0.trial169.stdout:607: oid 47 version 58 is already newer than 58 2026-02-25T08:44:36.323 INFO:tasks.rados.rados.0.trial169.stdout:607: oid 47 version 58 is already newer than 58 2026-02-25T08:44:36.323 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 47 v 58 (ObjNum 289 snap 79 seq_num 289) dirty exists 2026-02-25T08:44:36.323 INFO:tasks.rados.rados.0.trial169.stdout:607: left oid 47 (ObjNum 289 snap 79 seq_num 289) 2026-02-25T08:44:37.202 INFO:tasks.rados.rados.0.trial169.stdout:603: done (7 left) 2026-02-25T08:44:37.202 INFO:tasks.rados.rados.0.trial169.stdout:604: done (6 left) 2026-02-25T08:44:37.202 INFO:tasks.rados.rados.0.trial169.stdout:605: done (5 left) 2026-02-25T08:44:37.202 INFO:tasks.rados.rados.0.trial169.stdout:606: done (4 left) 2026-02-25T08:44:37.202 INFO:tasks.rados.rados.0.trial169.stdout:607: done (3 left) 2026-02-25T08:44:37.202 INFO:tasks.rados.rados.0.trial169.stdout:608: done (2 left) 2026-02-25T08:44:37.202 INFO:tasks.rados.rados.0.trial169.stdout:609: done (1 left) 2026-02-25T08:44:37.202 INFO:tasks.rados.rados.0.trial169.stdout:610: done (0 left) 2026-02-25T08:44:37.203 INFO:tasks.rados.rados.0.trial169.stdout:611: write oid 3 current snap is 80 2026-02-25T08:44:37.203 INFO:tasks.rados.rados.0.trial169.stdout:611: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2026-02-25T08:44:37.206 INFO:tasks.rados.rados.0.trial169.stdout:611: writing trial16941936-3 from 490837 to 1209710 tid 1 2026-02-25T08:44:37.210 INFO:tasks.rados.rados.0.trial169.stdout:611: writing trial16941936-3 from 1862260 to 2341687 tid 2 2026-02-25T08:44:37.213 INFO:tasks.rados.rados.0.trial169.stdout:611: writing trial16941936-3 from 2875032 to 2890837 tid 3 2026-02-25T08:44:37.213 INFO:tasks.rados.rados.0.trial169.stdout:612: read oid 50 snap 76 2026-02-25T08:44:37.213 INFO:tasks.rados.rados.0.trial169.stdout:612: expect (ObjNum 253 snap 72 seq_num 253) 2026-02-25T08:44:37.213 INFO:tasks.rados.rados.0.trial169.stdout:613: read oid 49 snap -1 2026-02-25T08:44:37.213 INFO:tasks.rados.rados.0.trial169.stdout:613: expect (ObjNum 290 snap 79 seq_num 290) 2026-02-25T08:44:37.214 INFO:tasks.rados.rados.0.trial169.stdout:614: read oid 43 snap -1 2026-02-25T08:44:37.214 INFO:tasks.rados.rados.0.trial169.stdout:614: expect (ObjNum 277 snap 76 seq_num 277) 2026-02-25T08:44:37.214 INFO:tasks.rados.rados.0.trial169.stdout:615: read oid 7 snap -1 2026-02-25T08:44:37.214 INFO:tasks.rados.rados.0.trial169.stdout:615: expect (ObjNum 255 snap 73 seq_num 255) 2026-02-25T08:44:37.214 INFO:tasks.rados.rados.0.trial169.stdout:616: delete oid 46 current snap is 80 2026-02-25T08:44:37.230 INFO:tasks.rados.rados.0.trial169.stdout:614: expect (ObjNum 277 snap 76 seq_num 277) 2026-02-25T08:44:37.252 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2622332 2026-02-25T08:44:37.257 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:37.307 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1507679 2026-02-25T08:44:37.326 INFO:tasks.rados.rados.0.trial169.stdout:614: done (5 left) 2026-02-25T08:44:37.327 INFO:tasks.rados.rados.0.trial169.stdout:616: done (4 left) 2026-02-25T08:44:37.327 INFO:tasks.rados.rados.0.trial169.stdout:617: write oid 37 current snap is 80 2026-02-25T08:44:37.327 INFO:tasks.rados.rados.0.trial169.stdout:617: seq_num 293 ranges {524451=469371,1579224=483096,2529878=394573} 2026-02-25T08:44:37.330 INFO:tasks.rados.rados.0.trial169.stdout:617: writing trial16941936-37 from 524451 to 993822 tid 1 2026-02-25T08:44:37.334 INFO:tasks.rados.rados.0.trial169.stdout:617: writing trial16941936-37 from 1579224 to 2062320 tid 2 2026-02-25T08:44:37.337 INFO:tasks.rados.rados.0.trial169.stdout:617: writing trial16941936-37 from 2529878 to 2924451 tid 3 2026-02-25T08:44:37.338 INFO:tasks.rados.rados.0.trial169.stdout:618: snap_remove snap 79 2026-02-25T08:44:38.141 INFO:tasks.rados.rados.0.trial169.stdout:618: done (5 left) 2026-02-25T08:44:38.142 INFO:tasks.rados.rados.0.trial169.stdout:619: write (excl) oid 35 current snap is 80 2026-02-25T08:44:38.142 INFO:tasks.rados.rados.0.trial169.stdout:613: expect (ObjNum 290 snap 79 seq_num 290) 2026-02-25T08:44:38.142 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 172698 2026-02-25T08:44:38.177 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:38.241 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:38.247 INFO:tasks.rados.rados.0.trial169.stdout:611: finishing write tid 1 to trial16941936-3 2026-02-25T08:44:38.247 INFO:tasks.rados.rados.0.trial169.stdout:615: expect (ObjNum 255 snap 73 seq_num 255) 2026-02-25T08:44:38.250 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1649494 2026-02-25T08:44:38.262 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1052262 2026-02-25T08:44:38.272 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 592193 2026-02-25T08:44:38.288 INFO:tasks.rados.rados.0.trial169.stdout:611: finishing write tid 2 to trial16941936-3 2026-02-25T08:44:38.288 INFO:tasks.rados.rados.0.trial169.stdout:611: finishing write tid 3 to trial16941936-3 2026-02-25T08:44:38.288 INFO:tasks.rados.rados.0.trial169.stdout:612: expect (ObjNum 253 snap 72 seq_num 253) 2026-02-25T08:44:38.316 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:38.313+0000 7fb7c7811640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:38.318 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:38.317+0000 7fb7c7811640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:38.322 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:38.355 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 202782 2026-02-25T08:44:38.362 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:38.362 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:38.390 INFO:tasks.rados.rados.0.trial169.stdout:611: finishing write tid 4 to trial16941936-3 2026-02-25T08:44:38.390 INFO:tasks.rados.rados.0.trial169.stdout:611: finishing write tid 5 to trial16941936-3 2026-02-25T08:44:38.390 INFO:tasks.rados.rados.0.trial169.stdout:611: oid 3 updating version 0 to 126 2026-02-25T08:44:38.390 INFO:tasks.rados.rados.0.trial169.stdout:611: oid 3 version 126 is already newer than 126 2026-02-25T08:44:38.390 INFO:tasks.rados.rados.0.trial169.stdout:611: oid 3 version 126 is already newer than 126 2026-02-25T08:44:38.390 INFO:tasks.rados.rados.0.trial169.stdout:611: oid 3 version 126 is already newer than 126 2026-02-25T08:44:38.390 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 3 v 126 (ObjNum 291 snap 80 seq_num 291) dirty exists 2026-02-25T08:44:38.390 INFO:tasks.rados.rados.0.trial169.stdout:611: left oid 3 (ObjNum 291 snap 80 seq_num 291) 2026-02-25T08:44:38.390 INFO:tasks.rados.rados.0.trial169.stdout:619: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2026-02-25T08:44:38.394 INFO:tasks.rados.rados.0.trial169.stdout:619: writing trial16941936-35 from 541258 to 1085878 tid 1 2026-02-25T08:44:38.399 INFO:tasks.rados.rados.0.trial169.stdout:619: writing trial16941936-35 from 1637706 to 2264460 tid 2 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:619: writing trial16941936-35 from 2840948 to 2941258 tid 3 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:611: done (5 left) 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:612: done (4 left) 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:613: done (3 left) 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:615: done (2 left) 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:620: snap_create 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:617: finishing write tid 1 to trial16941936-37 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:617: finishing write tid 2 to trial16941936-37 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:617: finishing write tid 3 to trial16941936-37 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:617: finishing write tid 4 to trial16941936-37 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:617: finishing write tid 5 to trial16941936-37 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:617: oid 37 updating version 0 to 110 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:617: oid 37 version 110 is already newer than 110 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:617: oid 37 version 110 is already newer than 110 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:617: oid 37 version 110 is already newer than 110 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 37 v 110 (ObjNum 293 snap 80 seq_num 293) dirty exists 2026-02-25T08:44:38.402 INFO:tasks.rados.rados.0.trial169.stdout:617: left oid 37 (ObjNum 293 snap 80 seq_num 293) 2026-02-25T08:44:38.408 INFO:tasks.rados.rados.0.trial169.stdout:619: finishing write tid 1 to trial16941936-35 2026-02-25T08:44:38.422 INFO:tasks.rados.rados.0.trial169.stdout:619: finishing write tid 2 to trial16941936-35 2026-02-25T08:44:38.425 INFO:tasks.rados.rados.0.trial169.stdout:619: finishing write tid 3 to trial16941936-35 2026-02-25T08:44:38.428 INFO:tasks.rados.rados.0.trial169.stdout:619: finishing write tid 4 to trial16941936-35 2026-02-25T08:44:38.430 INFO:tasks.rados.rados.0.trial169.stdout:619: finishing write tid 5 to trial16941936-35 2026-02-25T08:44:38.430 INFO:tasks.rados.rados.0.trial169.stdout:619: oid 35 updating version 0 to 331 2026-02-25T08:44:38.430 INFO:tasks.rados.rados.0.trial169.stdout:619: oid 35 version 331 is already newer than 328 2026-02-25T08:44:38.430 INFO:tasks.rados.rados.0.trial169.stdout:619: oid 35 version 331 is already newer than 331 2026-02-25T08:44:38.430 INFO:tasks.rados.rados.0.trial169.stdout:619: oid 35 version 331 is already newer than 331 2026-02-25T08:44:38.430 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 35 v 331 (ObjNum 294 snap 80 seq_num 294) dirty exists 2026-02-25T08:44:38.430 INFO:tasks.rados.rados.0.trial169.stdout:619: left oid 35 (ObjNum 294 snap 80 seq_num 294) 2026-02-25T08:44:39.209 INFO:tasks.rados.rados.0.trial169.stdout:617: done (2 left) 2026-02-25T08:44:39.209 INFO:tasks.rados.rados.0.trial169.stdout:619: done (1 left) 2026-02-25T08:44:39.209 INFO:tasks.rados.rados.0.trial169.stdout:620: done (0 left) 2026-02-25T08:44:39.209 INFO:tasks.rados.rados.0.trial169.stdout:621: write oid 49 current snap is 81 2026-02-25T08:44:39.209 INFO:tasks.rados.rados.0.trial169.stdout:621: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2026-02-25T08:44:39.213 INFO:tasks.rados.rados.0.trial169.stdout:621: writing trial16941936-49 from 558065 to 1177934 tid 1 2026-02-25T08:44:39.217 INFO:tasks.rados.rados.0.trial169.stdout:621: writing trial16941936-49 from 1579835 to 2066600 tid 2 2026-02-25T08:44:39.221 INFO:tasks.rados.rados.0.trial169.stdout:621: writing trial16941936-49 from 2468371 to 2958065 tid 3 2026-02-25T08:44:39.221 INFO:tasks.rados.rados.0.trial169.stdout:622: write oid 42 current snap is 81 2026-02-25T08:44:39.221 INFO:tasks.rados.rados.0.trial169.stdout:622: seq_num 296 ranges {574872=695118,2038317=630423} 2026-02-25T08:44:39.226 INFO:tasks.rados.rados.0.trial169.stdout:622: writing trial16941936-42 from 574872 to 1269990 tid 1 2026-02-25T08:44:39.231 INFO:tasks.rados.rados.0.trial169.stdout:622: writing trial16941936-42 from 2038317 to 2668740 tid 2 2026-02-25T08:44:39.231 INFO:tasks.rados.rados.0.trial169.stdout:623: snap_remove snap 77 2026-02-25T08:44:39.711 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:44:39.782 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:39.785 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:39.785 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:44:40.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~154s 2026-02-25T08:44:40.215 INFO:tasks.rados.rados.0.trial169.stdout:623: done (2 left) 2026-02-25T08:44:40.215 INFO:tasks.rados.rados.0.trial169.stdout:624: snap_create 2026-02-25T08:44:40.215 INFO:tasks.rados.rados.0.trial169.stdout:622: finishing write tid 1 to trial16941936-42 2026-02-25T08:44:40.215 INFO:tasks.rados.rados.0.trial169.stdout:622: finishing write tid 2 to trial16941936-42 2026-02-25T08:44:40.215 INFO:tasks.rados.rados.0.trial169.stdout:621: finishing write tid 1 to trial16941936-49 2026-02-25T08:44:40.215 INFO:tasks.rados.rados.0.trial169.stdout:622: finishing write tid 3 to trial16941936-42 2026-02-25T08:44:40.215 INFO:tasks.rados.rados.0.trial169.stdout:622: finishing write tid 4 to trial16941936-42 2026-02-25T08:44:40.216 INFO:tasks.rados.rados.0.trial169.stdout:622: oid 42 updating version 0 to 113 2026-02-25T08:44:40.216 INFO:tasks.rados.rados.0.trial169.stdout:622: oid 42 version 113 is already newer than 113 2026-02-25T08:44:40.216 INFO:tasks.rados.rados.0.trial169.stdout:622: oid 42 version 113 is already newer than 113 2026-02-25T08:44:40.216 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 42 v 113 (ObjNum 296 snap 81 seq_num 296) dirty exists 2026-02-25T08:44:40.216 INFO:tasks.rados.rados.0.trial169.stdout:622: left oid 42 (ObjNum 296 snap 81 seq_num 296) 2026-02-25T08:44:40.216 INFO:tasks.rados.rados.0.trial169.stdout:621: finishing write tid 2 to trial16941936-49 2026-02-25T08:44:40.216 INFO:tasks.rados.rados.0.trial169.stdout:621: finishing write tid 3 to trial16941936-49 2026-02-25T08:44:40.216 INFO:tasks.rados.rados.0.trial169.stdout:621: finishing write tid 4 to trial16941936-49 2026-02-25T08:44:40.216 INFO:tasks.rados.rados.0.trial169.stdout:621: finishing write tid 5 to trial16941936-49 2026-02-25T08:44:40.216 INFO:tasks.rados.rados.0.trial169.stdout:621: oid 49 updating version 0 to 267 2026-02-25T08:44:40.216 INFO:tasks.rados.rados.0.trial169.stdout:621: oid 49 version 267 is already newer than 267 2026-02-25T08:44:40.216 INFO:tasks.rados.rados.0.trial169.stdout:621: oid 49 version 267 is already newer than 267 2026-02-25T08:44:40.217 INFO:tasks.rados.rados.0.trial169.stdout:621: oid 49 version 267 is already newer than 267 2026-02-25T08:44:40.217 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 49 v 267 (ObjNum 295 snap 81 seq_num 295) dirty exists 2026-02-25T08:44:40.217 INFO:tasks.rados.rados.0.trial169.stdout:621: left oid 49 (ObjNum 295 snap 81 seq_num 295) 2026-02-25T08:44:40.364 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:40.440 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:40.437+0000 7f49967ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:40.442 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:40.441+0000 7f49967ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:40.446 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:40.486 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:40.486 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:41.217 INFO:tasks.rados.rados.0.trial169.stdout:621: done (2 left) 2026-02-25T08:44:41.217 INFO:tasks.rados.rados.0.trial169.stdout:622: done (1 left) 2026-02-25T08:44:41.217 INFO:tasks.rados.rados.0.trial169.stdout:624: done (0 left) 2026-02-25T08:44:41.217 INFO:tasks.rados.rados.0.trial169.stdout:625: read oid 7 snap -1 2026-02-25T08:44:41.218 INFO:tasks.rados.rados.0.trial169.stdout:625: expect (ObjNum 255 snap 73 seq_num 255) 2026-02-25T08:44:41.218 INFO:tasks.rados.rados.0.trial169.stdout:626: read oid 48 snap -1 2026-02-25T08:44:41.218 INFO:tasks.rados.rados.0.trial169.stdout:626: expect deleted 2026-02-25T08:44:41.218 INFO:tasks.rados.rados.0.trial169.stdout:627: snap_create 2026-02-25T08:44:41.251 INFO:tasks.rados.rados.0.trial169.stdout:625: expect (ObjNum 255 snap 73 seq_num 255) 2026-02-25T08:44:42.222 INFO:tasks.rados.rados.0.trial169.stdout:625: done (2 left) 2026-02-25T08:44:42.222 INFO:tasks.rados.rados.0.trial169.stdout:626: done (1 left) 2026-02-25T08:44:42.222 INFO:tasks.rados.rados.0.trial169.stdout:627: done (0 left) 2026-02-25T08:44:42.222 INFO:tasks.rados.rados.0.trial169.stdout:628: read oid 6 snap -1 2026-02-25T08:44:42.222 INFO:tasks.rados.rados.0.trial169.stdout:628: expect (ObjNum 288 snap 79 seq_num 288) 2026-02-25T08:44:42.222 INFO:tasks.rados.rados.0.trial169.stdout:629: write oid 23 current snap is 83 2026-02-25T08:44:42.222 INFO:tasks.rados.rados.0.trial169.stdout:629: seq_num 297 ranges {591679=486720,1813152=490434} 2026-02-25T08:44:42.226 INFO:tasks.rados.rados.0.trial169.stdout:629: writing trial16941936-23 from 591679 to 1078399 tid 1 2026-02-25T08:44:42.231 INFO:tasks.rados.rados.0.trial169.stdout:629: writing trial16941936-23 from 1813152 to 2303586 tid 2 2026-02-25T08:44:42.231 INFO:tasks.rados.rados.0.trial169.stdout:630: snap_create 2026-02-25T08:44:42.244 INFO:tasks.rados.rados.0.trial169.stdout:628: expect (ObjNum 288 snap 79 seq_num 288) 2026-02-25T08:44:42.288 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2382874 2026-02-25T08:44:42.293 INFO:tasks.rados.rados.0.trial169.stdout:629: finishing write tid 1 to trial16941936-23 2026-02-25T08:44:42.293 INFO:tasks.rados.rados.0.trial169.stdout:629: finishing write tid 2 to trial16941936-23 2026-02-25T08:44:42.293 INFO:tasks.rados.rados.0.trial169.stdout:629: finishing write tid 3 to trial16941936-23 2026-02-25T08:44:42.293 INFO:tasks.rados.rados.0.trial169.stdout:629: finishing write tid 4 to trial16941936-23 2026-02-25T08:44:42.293 INFO:tasks.rados.rados.0.trial169.stdout:629: oid 23 updating version 0 to 85 2026-02-25T08:44:42.293 INFO:tasks.rados.rados.0.trial169.stdout:629: oid 23 version 85 is already newer than 85 2026-02-25T08:44:42.293 INFO:tasks.rados.rados.0.trial169.stdout:629: oid 23 version 85 is already newer than 85 2026-02-25T08:44:42.293 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 23 v 85 (ObjNum 297 snap 83 seq_num 297) dirty exists 2026-02-25T08:44:42.293 INFO:tasks.rados.rados.0.trial169.stdout:629: left oid 23 (ObjNum 297 snap 83 seq_num 297) 2026-02-25T08:44:42.487 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:42.565 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:42.562+0000 7f3daa2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:42.567 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:42.566+0000 7f3daa2cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:42.571 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:42.612 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:43.227 INFO:tasks.rados.rados.0.trial169.stdout:628: done (2 left) 2026-02-25T08:44:43.227 INFO:tasks.rados.rados.0.trial169.stdout:629: done (1 left) 2026-02-25T08:44:43.227 INFO:tasks.rados.rados.0.trial169.stdout:630: done (0 left) 2026-02-25T08:44:43.228 INFO:tasks.rados.rados.0.trial169.stdout:631: read oid 11 snap -1 2026-02-25T08:44:43.228 INFO:tasks.rados.rados.0.trial169.stdout:631: expect deleted 2026-02-25T08:44:43.228 INFO:tasks.rados.rados.0.trial169.stdout:632: delete oid 35 current snap is 84 2026-02-25T08:44:43.261 INFO:tasks.rados.rados.0.trial169.stdout:631: done (1 left) 2026-02-25T08:44:43.262 INFO:tasks.rados.rados.0.trial169.stdout:632: done (0 left) 2026-02-25T08:44:43.262 INFO:tasks.rados.rados.0.trial169.stdout:633: snap_remove snap 80 2026-02-25T08:44:44.232 INFO:tasks.rados.rados.0.trial169.stdout:633: done (0 left) 2026-02-25T08:44:44.233 INFO:tasks.rados.rados.0.trial169.stdout:634: snap_remove snap 82 2026-02-25T08:44:44.613 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:44.690 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:44.687+0000 7fbe7941a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:44.693 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:44.691+0000 7fbe7941a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:44.696 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:44.735 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:44.735 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:44.786 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:44:44.855 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:44.858 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:44.859 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:44:45.235 INFO:tasks.rados.rados.0.trial169.stdout:634: done (0 left) 2026-02-25T08:44:45.235 INFO:tasks.rados.rados.0.trial169.stdout:635: read oid 33 snap 78 2026-02-25T08:44:45.235 INFO:tasks.rados.rados.0.trial169.stdout:635: expect deleted 2026-02-25T08:44:45.235 INFO:tasks.rados.rados.0.trial169.stdout:636: write oid 23 current snap is 84 2026-02-25T08:44:45.236 INFO:tasks.rados.rados.0.trial169.stdout:636: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2026-02-25T08:44:45.239 INFO:tasks.rados.rados.0.trial169.stdout:636: writing trial16941936-23 from 625293 to 1262511 tid 1 2026-02-25T08:44:45.243 INFO:tasks.rados.rados.0.trial169.stdout:636: writing trial16941936-23 from 1813763 to 2307866 tid 2 2026-02-25T08:44:45.246 INFO:tasks.rados.rados.0.trial169.stdout:636: writing trial16941936-23 from 2978063 to 3025293 tid 3 2026-02-25T08:44:45.247 INFO:tasks.rados.rados.0.trial169.stdout:637: delete oid 1 current snap is 84 2026-02-25T08:44:45.254 INFO:tasks.rados.rados.0.trial169.stdout:637: done (2 left) 2026-02-25T08:44:45.254 INFO:tasks.rados.rados.0.trial169.stdout:638: write oid 38 current snap is 84 2026-02-25T08:44:45.254 INFO:tasks.rados.rados.0.trial169.stdout:638: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2026-02-25T08:44:45.260 INFO:tasks.rados.rados.0.trial169.stdout:638: writing trial16941936-38 from 658907 to 1446623 tid 1 2026-02-25T08:44:45.263 INFO:tasks.rados.rados.0.trial169.stdout:638: writing trial16941936-38 from 1930727 to 2428499 tid 2 2026-02-25T08:44:45.266 INFO:tasks.rados.rados.0.trial169.stdout:638: writing trial16941936-38 from 3032909 to 3058907 tid 3 2026-02-25T08:44:45.266 INFO:tasks.rados.rados.0.trial169.stdout:639: write (excl) oid 20 current snap is 84 2026-02-25T08:44:45.266 INFO:tasks.rados.rados.0.trial169.stdout:639: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2026-02-25T08:44:45.270 INFO:tasks.rados.rados.0.trial169.stdout:639: writing trial16941936-20 from 675714 to 1138679 tid 1 2026-02-25T08:44:45.274 INFO:tasks.rados.rados.0.trial169.stdout:639: writing trial16941936-20 from 1589209 to 2230639 tid 2 2026-02-25T08:44:45.278 INFO:tasks.rados.rados.0.trial169.stdout:639: writing trial16941936-20 from 2943979 to 3075714 tid 3 2026-02-25T08:44:45.278 INFO:tasks.rados.rados.0.trial169.stdout:640: read oid 13 snap -1 2026-02-25T08:44:45.278 INFO:tasks.rados.rados.0.trial169.stdout:640: expect deleted 2026-02-25T08:44:45.278 INFO:tasks.rados.rados.0.trial169.stdout:641: snap_remove snap 74 2026-02-25T08:44:45.628 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~160s 2026-02-25T08:44:46.240 INFO:tasks.rados.rados.0.trial169.stdout:641: done (5 left) 2026-02-25T08:44:46.241 INFO:tasks.rados.rados.0.trial169.stdout:642: write oid 29 current snap is 84 2026-02-25T08:44:46.241 INFO:tasks.rados.rados.0.trial169.stdout:642: seq_num 303 ranges {692521=538214,1931338=785088} 2026-02-25T08:44:46.245 INFO:tasks.rados.rados.0.trial169.stdout:642: writing trial16941936-29 from 692521 to 1230735 tid 1 2026-02-25T08:44:46.253 INFO:tasks.rados.rados.0.trial169.stdout:642: writing trial16941936-29 from 1931338 to 2716426 tid 2 2026-02-25T08:44:46.253 INFO:tasks.rados.rados.0.trial169.stdout:635: done (5 left) 2026-02-25T08:44:46.253 INFO:tasks.rados.rados.0.trial169.stdout:643: read oid 34 snap -1 2026-02-25T08:44:46.253 INFO:tasks.rados.rados.0.trial169.stdout:643: expect (ObjNum 145 snap 32 seq_num 145) 2026-02-25T08:44:46.253 INFO:tasks.rados.rados.0.trial169.stdout:644: delete oid 17 current snap is 84 2026-02-25T08:44:46.253 INFO:tasks.rados.rados.0.trial169.stdout:636: finishing write tid 1 to trial16941936-23 2026-02-25T08:44:46.253 INFO:tasks.rados.rados.0.trial169.stdout:639: finishing write tid 1 to trial16941936-20 2026-02-25T08:44:46.254 INFO:tasks.rados.rados.0.trial169.stdout:636: finishing write tid 2 to trial16941936-23 2026-02-25T08:44:46.254 INFO:tasks.rados.rados.0.trial169.stdout:639: finishing write tid 2 to trial16941936-20 2026-02-25T08:44:46.254 INFO:tasks.rados.rados.0.trial169.stdout:636: finishing write tid 3 to trial16941936-23 2026-02-25T08:44:46.254 INFO:tasks.rados.rados.0.trial169.stdout:639: finishing write tid 3 to trial16941936-20 2026-02-25T08:44:46.254 INFO:tasks.rados.rados.0.trial169.stdout:636: finishing write tid 4 to trial16941936-23 2026-02-25T08:44:46.254 INFO:tasks.rados.rados.0.trial169.stdout:636: finishing write tid 5 to trial16941936-23 2026-02-25T08:44:46.254 INFO:tasks.rados.rados.0.trial169.stdout:636: oid 23 updating version 0 to 90 2026-02-25T08:44:46.254 INFO:tasks.rados.rados.0.trial169.stdout:636: oid 23 version 90 is already newer than 90 2026-02-25T08:44:46.254 INFO:tasks.rados.rados.0.trial169.stdout:636: oid 23 version 90 is already newer than 90 2026-02-25T08:44:46.254 INFO:tasks.rados.rados.0.trial169.stdout:636: oid 23 version 90 is already newer than 90 2026-02-25T08:44:46.254 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 23 v 90 (ObjNum 299 snap 84 seq_num 299) dirty exists 2026-02-25T08:44:46.254 INFO:tasks.rados.rados.0.trial169.stdout:636: left oid 23 (ObjNum 299 snap 84 seq_num 299) 2026-02-25T08:44:46.255 INFO:tasks.rados.rados.0.trial169.stdout:639: finishing write tid 4 to trial16941936-20 2026-02-25T08:44:46.255 INFO:tasks.rados.rados.0.trial169.stdout:639: finishing write tid 5 to trial16941936-20 2026-02-25T08:44:46.255 INFO:tasks.rados.rados.0.trial169.stdout:639: oid 20 updating version 0 to 101 2026-02-25T08:44:46.255 INFO:tasks.rados.rados.0.trial169.stdout:639: oid 20 updating version 101 to 104 2026-02-25T08:44:46.255 INFO:tasks.rados.rados.0.trial169.stdout:639: oid 20 version 104 is already newer than 104 2026-02-25T08:44:46.255 INFO:tasks.rados.rados.0.trial169.stdout:639: oid 20 version 104 is already newer than 104 2026-02-25T08:44:46.255 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 20 v 104 (ObjNum 302 snap 84 seq_num 302) dirty exists 2026-02-25T08:44:46.255 INFO:tasks.rados.rados.0.trial169.stdout:639: left oid 20 (ObjNum 302 snap 84 seq_num 302) 2026-02-25T08:44:46.255 INFO:tasks.rados.rados.0.trial169.stdout:638: finishing write tid 1 to trial16941936-38 2026-02-25T08:44:46.255 INFO:tasks.rados.rados.0.trial169.stdout:638: finishing write tid 2 to trial16941936-38 2026-02-25T08:44:46.255 INFO:tasks.rados.rados.0.trial169.stdout:638: finishing write tid 3 to trial16941936-38 2026-02-25T08:44:46.255 INFO:tasks.rados.rados.0.trial169.stdout:638: finishing write tid 4 to trial16941936-38 2026-02-25T08:44:46.256 INFO:tasks.rados.rados.0.trial169.stdout:638: finishing write tid 5 to trial16941936-38 2026-02-25T08:44:46.256 INFO:tasks.rados.rados.0.trial169.stdout:638: oid 38 updating version 0 to 63 2026-02-25T08:44:46.256 INFO:tasks.rados.rados.0.trial169.stdout:638: oid 38 version 63 is already newer than 63 2026-02-25T08:44:46.256 INFO:tasks.rados.rados.0.trial169.stdout:638: oid 38 version 63 is already newer than 63 2026-02-25T08:44:46.256 INFO:tasks.rados.rados.0.trial169.stdout:638: oid 38 version 63 is already newer than 63 2026-02-25T08:44:46.256 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 38 v 63 (ObjNum 301 snap 84 seq_num 301) dirty exists 2026-02-25T08:44:46.256 INFO:tasks.rados.rados.0.trial169.stdout:638: left oid 38 (ObjNum 301 snap 84 seq_num 301) 2026-02-25T08:44:46.287 INFO:tasks.rados.rados.0.trial169.stdout:636: done (6 left) 2026-02-25T08:44:46.287 INFO:tasks.rados.rados.0.trial169.stdout:638: done (5 left) 2026-02-25T08:44:46.287 INFO:tasks.rados.rados.0.trial169.stdout:639: done (4 left) 2026-02-25T08:44:46.287 INFO:tasks.rados.rados.0.trial169.stdout:640: done (3 left) 2026-02-25T08:44:46.287 INFO:tasks.rados.rados.0.trial169.stdout:644: done (2 left) 2026-02-25T08:44:46.287 INFO:tasks.rados.rados.0.trial169.stdout:645: delete oid 46 current snap is 84 2026-02-25T08:44:46.291 INFO:tasks.rados.rados.0.trial169.stdout:645: done (2 left) 2026-02-25T08:44:46.292 INFO:tasks.rados.rados.0.trial169.stdout:646: read oid 15 snap -1 2026-02-25T08:44:46.292 INFO:tasks.rados.rados.0.trial169.stdout:646: expect (ObjNum 191 snap 53 seq_num 191) 2026-02-25T08:44:46.292 INFO:tasks.rados.rados.0.trial169.stdout:647: read oid 32 snap -1 2026-02-25T08:44:46.292 INFO:tasks.rados.rados.0.trial169.stdout:647: expect (ObjNum 127 snap 21 seq_num 127) 2026-02-25T08:44:46.292 INFO:tasks.rados.rados.0.trial169.stdout:648: write (excl) oid 48 current snap is 84 2026-02-25T08:44:46.292 INFO:tasks.rados.rados.0.trial169.stdout:648: seq_num 306 ranges {742942=480314,1823137=648768} 2026-02-25T08:44:46.296 INFO:tasks.rados.rados.0.trial169.stdout:648: writing trial16941936-48 from 742942 to 1223256 tid 1 2026-02-25T08:44:46.301 INFO:tasks.rados.rados.0.trial169.stdout:648: writing trial16941936-48 from 1823137 to 2471905 tid 2 2026-02-25T08:44:46.301 INFO:tasks.rados.rados.0.trial169.stdout:649: snap_remove snap 81 2026-02-25T08:44:46.736 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:46.813 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:46.810+0000 7f92ccbec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:46.816 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:46.814+0000 7f92ccbec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:46.820 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:46.860 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:46.861 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:47.241 INFO:tasks.rados.rados.0.trial169.stdout:649: done (5 left) 2026-02-25T08:44:47.241 INFO:tasks.rados.rados.0.trial169.stdout:650: snap_remove snap 83 2026-02-25T08:44:48.144 INFO:tasks.rados.rados.0.trial169.stdout:650: done (5 left) 2026-02-25T08:44:48.144 INFO:tasks.rados.rados.0.trial169.stdout:651: read oid 17 snap -1 2026-02-25T08:44:48.144 INFO:tasks.rados.rados.0.trial169.stdout:651: expect deleted 2026-02-25T08:44:48.144 INFO:tasks.rados.rados.0.trial169.stdout:652: write oid 33 current snap is 84 2026-02-25T08:44:48.144 INFO:tasks.rados.rados.0.trial169.stdout:643: expect (ObjNum 145 snap 32 seq_num 145) 2026-02-25T08:44:48.145 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:48.226 INFO:tasks.rados.rados.0.trial169.stdout:652: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2026-02-25T08:44:48.232 INFO:tasks.rados.rados.0.trial169.stdout:652: writing trial16941936-33 from 759749 to 1315312 tid 1 2026-02-25T08:44:48.236 INFO:tasks.rados.rados.0.trial169.stdout:652: writing trial16941936-33 from 1765266 to 2557692 tid 2 2026-02-25T08:44:48.239 INFO:tasks.rados.rados.0.trial169.stdout:652: writing trial16941936-33 from 2964741 to 3159749 tid 3 2026-02-25T08:44:48.239 INFO:tasks.rados.rados.0.trial169.stdout:643: done (6 left) 2026-02-25T08:44:48.239 INFO:tasks.rados.rados.0.trial169.stdout:653: write (excl) oid 31 current snap is 84 2026-02-25T08:44:48.239 INFO:tasks.rados.rados.0.trial169.stdout:653: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2026-02-25T08:44:48.244 INFO:tasks.rados.rados.0.trial169.stdout:653: writing trial16941936-31 from 776556 to 1407368 tid 1 2026-02-25T08:44:48.248 INFO:tasks.rados.rados.0.trial169.stdout:653: writing trial16941936-31 from 1823748 to 2476185 tid 2 2026-02-25T08:44:48.251 INFO:tasks.rados.rados.0.trial169.stdout:653: writing trial16941936-31 from 3108517 to 3176556 tid 3 2026-02-25T08:44:48.251 INFO:tasks.rados.rados.0.trial169.stdout:654: snap_remove snap 78 2026-02-25T08:44:48.862 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:48.941 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:48.939+0000 7f07727ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:48.944 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:48.942+0000 7f07727ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:48.947 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:48.986 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:48.987 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:49.248 INFO:tasks.rados.rados.0.trial169.stdout:654: done (7 left) 2026-02-25T08:44:49.248 INFO:tasks.rados.rados.0.trial169.stdout:655: write (excl) oid 12 current snap is 84 2026-02-25T08:44:49.248 INFO:tasks.rados.rados.0.trial169.stdout:647: expect (ObjNum 127 snap 21 seq_num 127) 2026-02-25T08:44:49.276 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2121023 2026-02-25T08:44:49.293 INFO:tasks.rados.rados.0.trial169.stdout:646: expect (ObjNum 191 snap 53 seq_num 191) 2026-02-25T08:44:49.360 INFO:tasks.rados.rados.0.trial169.stdout:642: finishing write tid 1 to trial16941936-29 2026-02-25T08:44:49.360 INFO:tasks.rados.rados.0.trial169.stdout:648: finishing write tid 1 to trial16941936-48 2026-02-25T08:44:49.361 INFO:tasks.rados.rados.0.trial169.stdout:642: finishing write tid 2 to trial16941936-29 2026-02-25T08:44:49.361 INFO:tasks.rados.rados.0.trial169.stdout:648: finishing write tid 2 to trial16941936-48 2026-02-25T08:44:49.361 INFO:tasks.rados.rados.0.trial169.stdout:642: finishing write tid 3 to trial16941936-29 2026-02-25T08:44:49.361 INFO:tasks.rados.rados.0.trial169.stdout:655: seq_num 309 ranges {793363=706061,2282230=796095} 2026-02-25T08:44:49.366 INFO:tasks.rados.rados.0.trial169.stdout:655: writing trial16941936-12 from 793363 to 1499424 tid 1 2026-02-25T08:44:49.372 INFO:tasks.rados.rados.0.trial169.stdout:655: writing trial16941936-12 from 2282230 to 3078325 tid 2 2026-02-25T08:44:49.373 INFO:tasks.rados.rados.0.trial169.stdout:646: done (7 left) 2026-02-25T08:44:49.373 INFO:tasks.rados.rados.0.trial169.stdout:647: done (6 left) 2026-02-25T08:44:49.373 INFO:tasks.rados.rados.0.trial169.stdout:656: write (excl) oid 11 current snap is 84 2026-02-25T08:44:49.373 INFO:tasks.rados.rados.0.trial169.stdout:642: finishing write tid 4 to trial16941936-29 2026-02-25T08:44:49.373 INFO:tasks.rados.rados.0.trial169.stdout:642: oid 29 updating version 0 to 343 2026-02-25T08:44:49.373 INFO:tasks.rados.rados.0.trial169.stdout:642: oid 29 version 343 is already newer than 343 2026-02-25T08:44:49.373 INFO:tasks.rados.rados.0.trial169.stdout:642: oid 29 version 343 is already newer than 343 2026-02-25T08:44:49.373 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 29 v 343 (ObjNum 303 snap 84 seq_num 303) dirty exists 2026-02-25T08:44:49.373 INFO:tasks.rados.rados.0.trial169.stdout:642: left oid 29 (ObjNum 303 snap 84 seq_num 303) 2026-02-25T08:44:49.374 INFO:tasks.rados.rados.0.trial169.stdout:648: finishing write tid 3 to trial16941936-48 2026-02-25T08:44:49.374 INFO:tasks.rados.rados.0.trial169.stdout:648: finishing write tid 4 to trial16941936-48 2026-02-25T08:44:49.374 INFO:tasks.rados.rados.0.trial169.stdout:648: oid 48 updating version 0 to 20 2026-02-25T08:44:49.374 INFO:tasks.rados.rados.0.trial169.stdout:648: oid 48 version 20 is already newer than 18 2026-02-25T08:44:49.374 INFO:tasks.rados.rados.0.trial169.stdout:648: oid 48 version 20 is already newer than 20 2026-02-25T08:44:49.374 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 48 v 20 (ObjNum 306 snap 84 seq_num 306) dirty exists 2026-02-25T08:44:49.374 INFO:tasks.rados.rados.0.trial169.stdout:648: left oid 48 (ObjNum 306 snap 84 seq_num 306) 2026-02-25T08:44:49.374 INFO:tasks.rados.rados.0.trial169.stdout:656: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2026-02-25T08:44:49.376 INFO:tasks.rados.rados.0.trial169.stdout:656: writing trial16941936-11 from 410170 to 1191480 tid 1 2026-02-25T08:44:49.381 INFO:tasks.rados.rados.0.trial169.stdout:656: writing trial16941936-11 from 1940712 to 2596818 tid 2 2026-02-25T08:44:49.385 INFO:tasks.rados.rados.0.trial169.stdout:656: writing trial16941936-11 from 3163363 to 3210170 tid 3 2026-02-25T08:44:49.385 INFO:tasks.rados.rados.0.trial169.stdout:642: done (6 left) 2026-02-25T08:44:49.385 INFO:tasks.rados.rados.0.trial169.stdout:648: done (5 left) 2026-02-25T08:44:49.385 INFO:tasks.rados.rados.0.trial169.stdout:651: done (4 left) 2026-02-25T08:44:49.385 INFO:tasks.rados.rados.0.trial169.stdout:657: read oid 32 snap -1 2026-02-25T08:44:49.385 INFO:tasks.rados.rados.0.trial169.stdout:657: expect (ObjNum 127 snap 21 seq_num 127) 2026-02-25T08:44:49.385 INFO:tasks.rados.rados.0.trial169.stdout:658: delete oid 30 current snap is 84 2026-02-25T08:44:49.385 INFO:tasks.rados.rados.0.trial169.stdout:652: finishing write tid 1 to trial16941936-33 2026-02-25T08:44:49.386 INFO:tasks.rados.rados.0.trial169.stdout:652: finishing write tid 2 to trial16941936-33 2026-02-25T08:44:49.386 INFO:tasks.rados.rados.0.trial169.stdout:653: finishing write tid 1 to trial16941936-31 2026-02-25T08:44:49.386 INFO:tasks.rados.rados.0.trial169.stdout:652: finishing write tid 3 to trial16941936-33 2026-02-25T08:44:49.386 INFO:tasks.rados.rados.0.trial169.stdout:652: finishing write tid 4 to trial16941936-33 2026-02-25T08:44:49.386 INFO:tasks.rados.rados.0.trial169.stdout:652: finishing write tid 5 to trial16941936-33 2026-02-25T08:44:49.386 INFO:tasks.rados.rados.0.trial169.stdout:652: oid 33 updating version 0 to 349 2026-02-25T08:44:49.386 INFO:tasks.rados.rados.0.trial169.stdout:652: oid 33 version 349 is already newer than 349 2026-02-25T08:44:49.386 INFO:tasks.rados.rados.0.trial169.stdout:652: oid 33 version 349 is already newer than 349 2026-02-25T08:44:49.386 INFO:tasks.rados.rados.0.trial169.stdout:652: oid 33 version 349 is already newer than 349 2026-02-25T08:44:49.386 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 33 v 349 (ObjNum 307 snap 84 seq_num 307) dirty exists 2026-02-25T08:44:49.386 INFO:tasks.rados.rados.0.trial169.stdout:652: left oid 33 (ObjNum 307 snap 84 seq_num 307) 2026-02-25T08:44:49.386 INFO:tasks.rados.rados.0.trial169.stdout:653: finishing write tid 2 to trial16941936-31 2026-02-25T08:44:49.387 INFO:tasks.rados.rados.0.trial169.stdout:653: finishing write tid 3 to trial16941936-31 2026-02-25T08:44:49.387 INFO:tasks.rados.rados.0.trial169.stdout:653: finishing write tid 4 to trial16941936-31 2026-02-25T08:44:49.387 INFO:tasks.rados.rados.0.trial169.stdout:653: finishing write tid 5 to trial16941936-31 2026-02-25T08:44:49.387 INFO:tasks.rados.rados.0.trial169.stdout:653: oid 31 updating version 0 to 277 2026-02-25T08:44:49.387 INFO:tasks.rados.rados.0.trial169.stdout:653: oid 31 version 277 is already newer than 274 2026-02-25T08:44:49.387 INFO:tasks.rados.rados.0.trial169.stdout:653: oid 31 version 277 is already newer than 277 2026-02-25T08:44:49.387 INFO:tasks.rados.rados.0.trial169.stdout:653: oid 31 version 277 is already newer than 277 2026-02-25T08:44:49.387 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 31 v 277 (ObjNum 308 snap 84 seq_num 308) dirty exists 2026-02-25T08:44:49.387 INFO:tasks.rados.rados.0.trial169.stdout:653: left oid 31 (ObjNum 308 snap 84 seq_num 308) 2026-02-25T08:44:49.388 INFO:tasks.rados.rados.0.trial169.stdout:652: done (5 left) 2026-02-25T08:44:49.388 INFO:tasks.rados.rados.0.trial169.stdout:653: done (4 left) 2026-02-25T08:44:49.388 INFO:tasks.rados.rados.0.trial169.stdout:658: done (3 left) 2026-02-25T08:44:49.388 INFO:tasks.rados.rados.0.trial169.stdout:659: read oid 42 snap -1 2026-02-25T08:44:49.388 INFO:tasks.rados.rados.0.trial169.stdout:659: expect (ObjNum 296 snap 81 seq_num 296) 2026-02-25T08:44:49.388 INFO:tasks.rados.rados.0.trial169.stdout:660: write (excl) oid 43 current snap is 84 2026-02-25T08:44:49.389 INFO:tasks.rados.rados.0.trial169.stdout:660: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2026-02-25T08:44:49.391 INFO:tasks.rados.rados.0.trial169.stdout:660: writing trial16941936-43 from 443784 to 1091945 tid 1 2026-02-25T08:44:49.396 INFO:tasks.rados.rados.0.trial169.stdout:660: writing trial16941936-43 from 1657676 to 2317451 tid 2 2026-02-25T08:44:49.400 INFO:tasks.rados.rados.0.trial169.stdout:660: writing trial16941936-43 from 2818209 to 3243784 tid 3 2026-02-25T08:44:49.400 INFO:tasks.rados.rados.0.trial169.stdout:661: write (excl) oid 27 current snap is 84 2026-02-25T08:44:49.400 INFO:tasks.rados.rados.0.trial169.stdout:661: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2026-02-25T08:44:49.404 INFO:tasks.rados.rados.0.trial169.stdout:661: writing trial16941936-27 from 460591 to 1184001 tid 1 2026-02-25T08:44:49.407 INFO:tasks.rados.rados.0.trial169.stdout:661: writing trial16941936-27 from 1716158 to 2119591 tid 2 2026-02-25T08:44:49.412 INFO:tasks.rados.rados.0.trial169.stdout:661: writing trial16941936-27 from 2729279 to 3169419 tid 3 2026-02-25T08:44:49.412 INFO:tasks.rados.rados.0.trial169.stdout:662: snap_create 2026-02-25T08:44:49.412 INFO:tasks.rados.rados.0.trial169.stdout:655: finishing write tid 1 to trial16941936-12 2026-02-25T08:44:49.412 INFO:tasks.rados.rados.0.trial169.stdout:659: expect (ObjNum 296 snap 81 seq_num 296) 2026-02-25T08:44:49.438 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:49.463 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 122016 2026-02-25T08:44:49.488 INFO:tasks.rados.rados.0.trial169.stdout:660: finishing write tid 1 to trial16941936-43 2026-02-25T08:44:49.488 INFO:tasks.rados.rados.0.trial169.stdout:655: finishing write tid 2 to trial16941936-12 2026-02-25T08:44:49.488 INFO:tasks.rados.rados.0.trial169.stdout:655: finishing write tid 3 to trial16941936-12 2026-02-25T08:44:49.488 INFO:tasks.rados.rados.0.trial169.stdout:657: expect (ObjNum 127 snap 21 seq_num 127) 2026-02-25T08:44:49.526 INFO:tasks.rados.rados.0.trial169.stdout:660: finishing write tid 2 to trial16941936-43 2026-02-25T08:44:49.526 INFO:tasks.rados.rados.0.trial169.stdout:655: finishing write tid 4 to trial16941936-12 2026-02-25T08:44:49.527 INFO:tasks.rados.rados.0.trial169.stdout:655: oid 12 updating version 0 to 67 2026-02-25T08:44:49.527 INFO:tasks.rados.rados.0.trial169.stdout:655: oid 12 version 67 is already newer than 67 2026-02-25T08:44:49.527 INFO:tasks.rados.rados.0.trial169.stdout:655: oid 12 version 67 is already newer than 65 2026-02-25T08:44:49.527 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 12 v 67 (ObjNum 309 snap 84 seq_num 309) dirty exists 2026-02-25T08:44:49.527 INFO:tasks.rados.rados.0.trial169.stdout:655: left oid 12 (ObjNum 309 snap 84 seq_num 309) 2026-02-25T08:44:49.527 INFO:tasks.rados.rados.0.trial169.stdout:660: finishing write tid 3 to trial16941936-43 2026-02-25T08:44:49.527 INFO:tasks.rados.rados.0.trial169.stdout:656: finishing write tid 1 to trial16941936-11 2026-02-25T08:44:49.527 INFO:tasks.rados.rados.0.trial169.stdout:660: finishing write tid 4 to trial16941936-43 2026-02-25T08:44:49.527 INFO:tasks.rados.rados.0.trial169.stdout:660: finishing write tid 5 to trial16941936-43 2026-02-25T08:44:49.527 INFO:tasks.rados.rados.0.trial169.stdout:660: oid 43 updating version 0 to 51 2026-02-25T08:44:49.527 INFO:tasks.rados.rados.0.trial169.stdout:660: oid 43 version 51 is already newer than 51 2026-02-25T08:44:49.528 INFO:tasks.rados.rados.0.trial169.stdout:660: oid 43 version 51 is already newer than 51 2026-02-25T08:44:49.528 INFO:tasks.rados.rados.0.trial169.stdout:660: oid 43 version 51 is already newer than 48 2026-02-25T08:44:49.528 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 43 v 51 (ObjNum 312 snap 84 seq_num 312) dirty exists 2026-02-25T08:44:49.528 INFO:tasks.rados.rados.0.trial169.stdout:660: left oid 43 (ObjNum 312 snap 84 seq_num 312) 2026-02-25T08:44:49.528 INFO:tasks.rados.rados.0.trial169.stdout:661: finishing write tid 1 to trial16941936-27 2026-02-25T08:44:49.528 INFO:tasks.rados.rados.0.trial169.stdout:656: finishing write tid 2 to trial16941936-11 2026-02-25T08:44:49.528 INFO:tasks.rados.rados.0.trial169.stdout:656: finishing write tid 3 to trial16941936-11 2026-02-25T08:44:49.528 INFO:tasks.rados.rados.0.trial169.stdout:656: finishing write tid 4 to trial16941936-11 2026-02-25T08:44:49.528 INFO:tasks.rados.rados.0.trial169.stdout:661: finishing write tid 2 to trial16941936-27 2026-02-25T08:44:49.528 INFO:tasks.rados.rados.0.trial169.stdout:656: finishing write tid 5 to trial16941936-11 2026-02-25T08:44:49.528 INFO:tasks.rados.rados.0.trial169.stdout:656: oid 11 updating version 0 to 41 2026-02-25T08:44:49.528 INFO:tasks.rados.rados.0.trial169.stdout:656: oid 11 updating version 41 to 44 2026-02-25T08:44:49.528 INFO:tasks.rados.rados.0.trial169.stdout:656: oid 11 version 44 is already newer than 44 2026-02-25T08:44:49.529 INFO:tasks.rados.rados.0.trial169.stdout:656: oid 11 version 44 is already newer than 44 2026-02-25T08:44:49.529 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 11 v 44 (ObjNum 310 snap 84 seq_num 310) dirty exists 2026-02-25T08:44:49.529 INFO:tasks.rados.rados.0.trial169.stdout:656: left oid 11 (ObjNum 310 snap 84 seq_num 310) 2026-02-25T08:44:49.529 INFO:tasks.rados.rados.0.trial169.stdout:661: finishing write tid 3 to trial16941936-27 2026-02-25T08:44:49.529 INFO:tasks.rados.rados.0.trial169.stdout:661: finishing write tid 4 to trial16941936-27 2026-02-25T08:44:49.529 INFO:tasks.rados.rados.0.trial169.stdout:661: finishing write tid 5 to trial16941936-27 2026-02-25T08:44:49.529 INFO:tasks.rados.rados.0.trial169.stdout:661: oid 27 updating version 0 to 55 2026-02-25T08:44:49.529 INFO:tasks.rados.rados.0.trial169.stdout:661: oid 27 version 55 is already newer than 52 2026-02-25T08:44:49.529 INFO:tasks.rados.rados.0.trial169.stdout:661: oid 27 version 55 is already newer than 55 2026-02-25T08:44:49.529 INFO:tasks.rados.rados.0.trial169.stdout:661: oid 27 version 55 is already newer than 55 2026-02-25T08:44:49.529 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 27 v 55 (ObjNum 313 snap 84 seq_num 313) dirty exists 2026-02-25T08:44:49.529 INFO:tasks.rados.rados.0.trial169.stdout:661: left oid 27 (ObjNum 313 snap 84 seq_num 313) 2026-02-25T08:44:49.860 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:44:49.931 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:49.934 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:49.935 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:44:50.254 INFO:tasks.rados.rados.0.trial169.stdout:655: done (6 left) 2026-02-25T08:44:50.255 INFO:tasks.rados.rados.0.trial169.stdout:656: done (5 left) 2026-02-25T08:44:50.255 INFO:tasks.rados.rados.0.trial169.stdout:657: done (4 left) 2026-02-25T08:44:50.255 INFO:tasks.rados.rados.0.trial169.stdout:659: done (3 left) 2026-02-25T08:44:50.255 INFO:tasks.rados.rados.0.trial169.stdout:660: done (2 left) 2026-02-25T08:44:50.255 INFO:tasks.rados.rados.0.trial169.stdout:661: done (1 left) 2026-02-25T08:44:50.255 INFO:tasks.rados.rados.0.trial169.stdout:662: done (0 left) 2026-02-25T08:44:50.255 INFO:tasks.rados.rados.0.trial169.stdout:663: read oid 48 snap -1 2026-02-25T08:44:50.255 INFO:tasks.rados.rados.0.trial169.stdout:663: expect (ObjNum 306 snap 84 seq_num 306) 2026-02-25T08:44:50.255 INFO:tasks.rados.rados.0.trial169.stdout:664: write oid 26 current snap is 85 2026-02-25T08:44:50.256 INFO:tasks.rados.rados.0.trial169.stdout:664: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2026-02-25T08:44:50.259 INFO:tasks.rados.rados.0.trial169.stdout:664: writing trial16941936-26 from 477398 to 1276057 tid 1 2026-02-25T08:44:50.263 INFO:tasks.rados.rados.0.trial169.stdout:664: writing trial16941936-26 from 1774640 to 2438084 tid 2 2026-02-25T08:44:50.267 INFO:tasks.rados.rados.0.trial169.stdout:664: writing trial16941936-26 from 2873055 to 3277398 tid 3 2026-02-25T08:44:50.267 INFO:tasks.rados.rados.0.trial169.stdout:665: write (excl) oid 27 current snap is 85 2026-02-25T08:44:50.267 INFO:tasks.rados.rados.0.trial169.stdout:665: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2026-02-25T08:44:50.270 INFO:tasks.rados.rados.0.trial169.stdout:665: writing trial16941936-27 from 494205 to 968113 tid 1 2026-02-25T08:44:50.273 INFO:tasks.rados.rados.0.trial169.stdout:665: writing trial16941936-27 from 1433122 to 1840224 tid 2 2026-02-25T08:44:50.277 INFO:tasks.rados.rados.0.trial169.stdout:665: writing trial16941936-27 from 2384125 to 2846809 tid 3 2026-02-25T08:44:50.278 INFO:tasks.rados.rados.0.trial169.stdout:666: write (excl) oid 19 current snap is 85 2026-02-25T08:44:50.278 INFO:tasks.rados.rados.0.trial169.stdout:666: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2026-02-25T08:44:50.281 INFO:tasks.rados.rados.0.trial169.stdout:666: writing trial16941936-19 from 511012 to 1060169 tid 1 2026-02-25T08:44:50.286 INFO:tasks.rados.rados.0.trial169.stdout:666: writing trial16941936-19 from 1775251 to 2326011 tid 2 2026-02-25T08:44:50.290 INFO:tasks.rados.rados.0.trial169.stdout:666: writing trial16941936-19 from 3095195 to 3311012 tid 3 2026-02-25T08:44:50.290 INFO:tasks.rados.rados.0.trial169.stdout:667: delete oid 21 current snap is 85 2026-02-25T08:44:50.290 INFO:tasks.rados.rados.0.trial169.stdout:663: expect (ObjNum 306 snap 84 seq_num 306) 2026-02-25T08:44:50.317 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:50.361 INFO:tasks.rados.rados.0.trial169.stdout:666: finishing write tid 1 to trial16941936-19 2026-02-25T08:44:50.362 INFO:tasks.rados.rados.0.trial169.stdout:665: finishing write tid 1 to trial16941936-27 2026-02-25T08:44:50.362 INFO:tasks.rados.rados.0.trial169.stdout:666: finishing write tid 2 to trial16941936-19 2026-02-25T08:44:50.362 INFO:tasks.rados.rados.0.trial169.stdout:665: finishing write tid 2 to trial16941936-27 2026-02-25T08:44:50.362 INFO:tasks.rados.rados.0.trial169.stdout:666: finishing write tid 3 to trial16941936-19 2026-02-25T08:44:50.362 INFO:tasks.rados.rados.0.trial169.stdout:663: done (4 left) 2026-02-25T08:44:50.362 INFO:tasks.rados.rados.0.trial169.stdout:667: done (3 left) 2026-02-25T08:44:50.362 INFO:tasks.rados.rados.0.trial169.stdout:668: delete oid 32 current snap is 85 2026-02-25T08:44:50.362 INFO:tasks.rados.rados.0.trial169.stdout:666: finishing write tid 4 to trial16941936-19 2026-02-25T08:44:50.362 INFO:tasks.rados.rados.0.trial169.stdout:666: finishing write tid 5 to trial16941936-19 2026-02-25T08:44:50.363 INFO:tasks.rados.rados.0.trial169.stdout:666: oid 19 updating version 0 to 94 2026-02-25T08:44:50.363 INFO:tasks.rados.rados.0.trial169.stdout:666: oid 19 version 94 is already newer than 94 2026-02-25T08:44:50.363 INFO:tasks.rados.rados.0.trial169.stdout:666: oid 19 version 94 is already newer than 94 2026-02-25T08:44:50.363 INFO:tasks.rados.rados.0.trial169.stdout:666: oid 19 version 94 is already newer than 91 2026-02-25T08:44:50.363 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 19 v 94 (ObjNum 316 snap 85 seq_num 316) dirty exists 2026-02-25T08:44:50.363 INFO:tasks.rados.rados.0.trial169.stdout:666: left oid 19 (ObjNum 316 snap 85 seq_num 316) 2026-02-25T08:44:50.363 INFO:tasks.rados.rados.0.trial169.stdout:665: finishing write tid 3 to trial16941936-27 2026-02-25T08:44:50.363 INFO:tasks.rados.rados.0.trial169.stdout:665: finishing write tid 4 to trial16941936-27 2026-02-25T08:44:50.363 INFO:tasks.rados.rados.0.trial169.stdout:665: finishing write tid 5 to trial16941936-27 2026-02-25T08:44:50.363 INFO:tasks.rados.rados.0.trial169.stdout:665: oid 27 updating version 0 to 60 2026-02-25T08:44:50.363 INFO:tasks.rados.rados.0.trial169.stdout:665: oid 27 version 60 is already newer than 60 2026-02-25T08:44:50.363 INFO:tasks.rados.rados.0.trial169.stdout:665: oid 27 version 60 is already newer than 60 2026-02-25T08:44:50.363 INFO:tasks.rados.rados.0.trial169.stdout:665: oid 27 version 60 is already newer than 57 2026-02-25T08:44:50.364 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 27 v 60 (ObjNum 315 snap 85 seq_num 315) dirty exists 2026-02-25T08:44:50.364 INFO:tasks.rados.rados.0.trial169.stdout:665: left oid 27 (ObjNum 315 snap 85 seq_num 315) 2026-02-25T08:44:50.364 INFO:tasks.rados.rados.0.trial169.stdout:664: finishing write tid 1 to trial16941936-26 2026-02-25T08:44:50.364 INFO:tasks.rados.rados.0.trial169.stdout:664: finishing write tid 2 to trial16941936-26 2026-02-25T08:44:50.364 INFO:tasks.rados.rados.0.trial169.stdout:664: finishing write tid 3 to trial16941936-26 2026-02-25T08:44:50.364 INFO:tasks.rados.rados.0.trial169.stdout:664: finishing write tid 4 to trial16941936-26 2026-02-25T08:44:50.364 INFO:tasks.rados.rados.0.trial169.stdout:664: finishing write tid 5 to trial16941936-26 2026-02-25T08:44:50.364 INFO:tasks.rados.rados.0.trial169.stdout:664: oid 26 updating version 0 to 284 2026-02-25T08:44:50.364 INFO:tasks.rados.rados.0.trial169.stdout:664: oid 26 version 284 is already newer than 284 2026-02-25T08:44:50.364 INFO:tasks.rados.rados.0.trial169.stdout:664: oid 26 version 284 is already newer than 284 2026-02-25T08:44:50.364 INFO:tasks.rados.rados.0.trial169.stdout:664: oid 26 version 284 is already newer than 284 2026-02-25T08:44:50.364 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 26 v 284 (ObjNum 314 snap 85 seq_num 314) dirty exists 2026-02-25T08:44:50.364 INFO:tasks.rados.rados.0.trial169.stdout:664: left oid 26 (ObjNum 314 snap 85 seq_num 314) 2026-02-25T08:44:50.385 INFO:tasks.rados.rados.0.trial169.stdout:664: done (3 left) 2026-02-25T08:44:50.385 INFO:tasks.rados.rados.0.trial169.stdout:665: done (2 left) 2026-02-25T08:44:50.385 INFO:tasks.rados.rados.0.trial169.stdout:666: done (1 left) 2026-02-25T08:44:50.386 INFO:tasks.rados.rados.0.trial169.stdout:668: done (0 left) 2026-02-25T08:44:50.386 INFO:tasks.rados.rados.0.trial169.stdout:669: snap_create 2026-02-25T08:44:50.987 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:51.065 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:51.062+0000 7feebdc1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:51.067 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:51.066+0000 7feebdc1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:51.071 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:51.110 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:51.111 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:51.132 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~165s 2026-02-25T08:44:51.256 INFO:tasks.rados.rados.0.trial169.stdout:669: done (0 left) 2026-02-25T08:44:51.256 INFO:tasks.rados.rados.0.trial169.stdout:670: read oid 42 snap -1 2026-02-25T08:44:51.257 INFO:tasks.rados.rados.0.trial169.stdout:670: expect (ObjNum 296 snap 81 seq_num 296) 2026-02-25T08:44:51.257 INFO:tasks.rados.rados.0.trial169.stdout:671: write oid 28 current snap is 86 2026-02-25T08:44:51.257 INFO:tasks.rados.rados.0.trial169.stdout:671: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2026-02-25T08:44:51.261 INFO:tasks.rados.rados.0.trial169.stdout:671: writing trial16941936-28 from 561433 to 1336337 tid 1 2026-02-25T08:44:51.265 INFO:tasks.rados.rados.0.trial169.stdout:671: writing trial16941936-28 from 1950697 to 2365137 tid 2 2026-02-25T08:44:51.270 INFO:tasks.rados.rados.0.trial169.stdout:671: writing trial16941936-28 from 2777464 to 3361433 tid 3 2026-02-25T08:44:51.270 INFO:tasks.rados.rados.0.trial169.stdout:672: snap_remove snap 84 2026-02-25T08:44:52.262 INFO:tasks.rados.rados.0.trial169.stdout:672: done (2 left) 2026-02-25T08:44:52.262 INFO:tasks.rados.rados.0.trial169.stdout:673: write (excl) oid 29 current snap is 86 2026-02-25T08:44:52.262 INFO:tasks.rados.rados.0.trial169.stdout:673: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2026-02-25T08:44:52.266 INFO:tasks.rados.rados.0.trial169.stdout:673: writing trial16941936-29 from 578240 to 1144746 tid 1 2026-02-25T08:44:52.270 INFO:tasks.rados.rados.0.trial169.stdout:673: writing trial16941936-29 from 1609179 to 2167277 tid 2 2026-02-25T08:44:52.276 INFO:tasks.rados.rados.0.trial169.stdout:673: writing trial16941936-29 from 2804887 to 3356637 tid 3 2026-02-25T08:44:52.276 INFO:tasks.rados.rados.0.trial169.stdout:674: snap_create 2026-02-25T08:44:52.276 INFO:tasks.rados.rados.0.trial169.stdout:670: expect (ObjNum 296 snap 81 seq_num 296) 2026-02-25T08:44:52.306 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:44:52.331 INFO:tasks.rados.rados.0.trial169.stdout:671: finishing write tid 1 to trial16941936-28 2026-02-25T08:44:52.332 INFO:tasks.rados.rados.0.trial169.stdout:671: finishing write tid 2 to trial16941936-28 2026-02-25T08:44:52.332 INFO:tasks.rados.rados.0.trial169.stdout:671: finishing write tid 3 to trial16941936-28 2026-02-25T08:44:52.332 INFO:tasks.rados.rados.0.trial169.stdout:671: finishing write tid 4 to trial16941936-28 2026-02-25T08:44:52.332 INFO:tasks.rados.rados.0.trial169.stdout:671: finishing write tid 5 to trial16941936-28 2026-02-25T08:44:52.332 INFO:tasks.rados.rados.0.trial169.stdout:671: oid 28 updating version 0 to 49 2026-02-25T08:44:52.332 INFO:tasks.rados.rados.0.trial169.stdout:671: oid 28 version 49 is already newer than 49 2026-02-25T08:44:52.332 INFO:tasks.rados.rados.0.trial169.stdout:671: oid 28 version 49 is already newer than 49 2026-02-25T08:44:52.332 INFO:tasks.rados.rados.0.trial169.stdout:671: oid 28 version 49 is already newer than 49 2026-02-25T08:44:52.332 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 28 v 49 (ObjNum 319 snap 86 seq_num 319) dirty exists 2026-02-25T08:44:52.332 INFO:tasks.rados.rados.0.trial169.stdout:671: left oid 28 (ObjNum 319 snap 86 seq_num 319) 2026-02-25T08:44:52.333 INFO:tasks.rados.rados.0.trial169.stdout:673: finishing write tid 1 to trial16941936-29 2026-02-25T08:44:52.333 INFO:tasks.rados.rados.0.trial169.stdout:673: finishing write tid 2 to trial16941936-29 2026-02-25T08:44:52.333 INFO:tasks.rados.rados.0.trial169.stdout:673: finishing write tid 3 to trial16941936-29 2026-02-25T08:44:52.333 INFO:tasks.rados.rados.0.trial169.stdout:673: finishing write tid 4 to trial16941936-29 2026-02-25T08:44:52.333 INFO:tasks.rados.rados.0.trial169.stdout:673: finishing write tid 5 to trial16941936-29 2026-02-25T08:44:52.333 INFO:tasks.rados.rados.0.trial169.stdout:673: oid 29 updating version 0 to 360 2026-02-25T08:44:52.334 INFO:tasks.rados.rados.0.trial169.stdout:673: oid 29 version 360 is already newer than 357 2026-02-25T08:44:52.334 INFO:tasks.rados.rados.0.trial169.stdout:673: oid 29 version 360 is already newer than 360 2026-02-25T08:44:52.334 INFO:tasks.rados.rados.0.trial169.stdout:673: oid 29 version 360 is already newer than 360 2026-02-25T08:44:52.334 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 29 v 360 (ObjNum 320 snap 86 seq_num 320) dirty exists 2026-02-25T08:44:52.334 INFO:tasks.rados.rados.0.trial169.stdout:673: left oid 29 (ObjNum 320 snap 86 seq_num 320) 2026-02-25T08:44:53.112 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:53.187 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:53.184+0000 7f1a88011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:53.189 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:53.188+0000 7f1a88011640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:53.193 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:53.232 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:53.233 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:53.264 INFO:tasks.rados.rados.0.trial169.stdout:670: done (3 left) 2026-02-25T08:44:53.265 INFO:tasks.rados.rados.0.trial169.stdout:671: done (2 left) 2026-02-25T08:44:53.265 INFO:tasks.rados.rados.0.trial169.stdout:673: done (1 left) 2026-02-25T08:44:53.265 INFO:tasks.rados.rados.0.trial169.stdout:674: done (0 left) 2026-02-25T08:44:53.265 INFO:tasks.rados.rados.0.trial169.stdout:675: snap_remove snap 86 2026-02-25T08:44:54.267 INFO:tasks.rados.rados.0.trial169.stdout:675: done (0 left) 2026-02-25T08:44:54.267 INFO:tasks.rados.rados.0.trial169.stdout:676: read oid 12 snap -1 2026-02-25T08:44:54.267 INFO:tasks.rados.rados.0.trial169.stdout:676: expect (ObjNum 309 snap 84 seq_num 309) 2026-02-25T08:44:54.268 INFO:tasks.rados.rados.0.trial169.stdout:677: delete oid 35 current snap is 87 2026-02-25T08:44:54.294 INFO:tasks.rados.rados.0.trial169.stdout:677: done (1 left) 2026-02-25T08:44:54.294 INFO:tasks.rados.rados.0.trial169.stdout:678: read oid 39 snap -1 2026-02-25T08:44:54.294 INFO:tasks.rados.rados.0.trial169.stdout:678: expect deleted 2026-02-25T08:44:54.294 INFO:tasks.rados.rados.0.trial169.stdout:679: write oid 34 current snap is 87 2026-02-25T08:44:54.294 INFO:tasks.rados.rados.0.trial169.stdout:679: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2026-02-25T08:44:54.299 INFO:tasks.rados.rados.0.trial169.stdout:679: writing trial16941936-34 from 611854 to 1328858 tid 1 2026-02-25T08:44:54.304 INFO:tasks.rados.rados.0.trial169.stdout:679: writing trial16941936-34 from 2126143 to 2687910 tid 2 2026-02-25T08:44:54.307 INFO:tasks.rados.rados.0.trial169.stdout:679: writing trial16941936-34 from 3259733 to 3411854 tid 3 2026-02-25T08:44:54.307 INFO:tasks.rados.rados.0.trial169.stdout:680: read oid 4 snap -1 2026-02-25T08:44:54.307 INFO:tasks.rados.rados.0.trial169.stdout:680: expect (ObjNum 278 snap 76 seq_num 278) 2026-02-25T08:44:54.307 INFO:tasks.rados.rados.0.trial169.stdout:681: snap_remove snap 85 2026-02-25T08:44:54.936 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:44:55.008 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:44:55.011 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:44:55.011 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:44:55.234 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:55.270 INFO:tasks.rados.rados.0.trial169.stdout:681: done (4 left) 2026-02-25T08:44:55.271 INFO:tasks.rados.rados.0.trial169.stdout:682: snap_remove snap 76 2026-02-25T08:44:55.310 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:55.308+0000 7ff2901b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:55.313 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:55.311+0000 7ff2901b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:55.316 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:55.354 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:55.355 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:56.275 INFO:tasks.rados.rados.0.trial169.stdout:682: done (4 left) 2026-02-25T08:44:56.276 INFO:tasks.rados.rados.0.trial169.stdout:683: write oid 18 current snap is 87 2026-02-25T08:44:56.276 INFO:tasks.rados.rados.0.trial169.stdout:683: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2026-02-25T08:44:56.281 INFO:tasks.rados.rados.0.trial169.stdout:683: writing trial16941936-18 from 628661 to 1420914 tid 1 2026-02-25T08:44:56.285 INFO:tasks.rados.rados.0.trial169.stdout:683: writing trial16941936-18 from 2184625 to 2606403 tid 2 2026-02-25T08:44:56.288 INFO:tasks.rados.rados.0.trial169.stdout:683: writing trial16941936-18 from 3403509 to 3428661 tid 3 2026-02-25T08:44:56.289 INFO:tasks.rados.rados.0.trial169.stdout:684: write (excl) oid 42 current snap is 87 2026-02-25T08:44:56.289 INFO:tasks.rados.rados.0.trial169.stdout:684: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2026-02-25T08:44:56.292 INFO:tasks.rados.rados.0.trial169.stdout:684: writing trial16941936-42 from 645468 to 1112970 tid 1 2026-02-25T08:44:56.297 INFO:tasks.rados.rados.0.trial169.stdout:684: writing trial16941936-42 from 1726754 to 2292190 tid 2 2026-02-25T08:44:56.302 INFO:tasks.rados.rados.0.trial169.stdout:684: writing trial16941936-42 from 2798226 to 3395064 tid 3 2026-02-25T08:44:56.302 INFO:tasks.rados.rados.0.trial169.stdout:685: read oid 36 snap -1 2026-02-25T08:44:56.302 INFO:tasks.rados.rados.0.trial169.stdout:685: expect deleted 2026-02-25T08:44:56.302 INFO:tasks.rados.rados.0.trial169.stdout:686: read oid 28 snap -1 2026-02-25T08:44:56.302 INFO:tasks.rados.rados.0.trial169.stdout:686: expect (ObjNum 319 snap 86 seq_num 319) 2026-02-25T08:44:56.302 INFO:tasks.rados.rados.0.trial169.stdout:687: snap_create 2026-02-25T08:44:56.302 INFO:tasks.rados.rados.0.trial169.stdout:676: expect (ObjNum 309 snap 84 seq_num 309) 2026-02-25T08:44:56.396 INFO:tasks.rados.rados.0.trial169.stdout:680: expect (ObjNum 278 snap 76 seq_num 278) 2026-02-25T08:44:56.464 INFO:tasks.rados.rados.0.trial169.stdout:679: finishing write tid 1 to trial16941936-34 2026-02-25T08:44:56.465 INFO:tasks.rados.rados.0.trial169.stdout:679: finishing write tid 2 to trial16941936-34 2026-02-25T08:44:56.465 INFO:tasks.rados.rados.0.trial169.stdout:679: finishing write tid 3 to trial16941936-34 2026-02-25T08:44:56.465 INFO:tasks.rados.rados.0.trial169.stdout:679: finishing write tid 4 to trial16941936-34 2026-02-25T08:44:56.465 INFO:tasks.rados.rados.0.trial169.stdout:679: finishing write tid 5 to trial16941936-34 2026-02-25T08:44:56.465 INFO:tasks.rados.rados.0.trial169.stdout:679: oid 34 updating version 0 to 99 2026-02-25T08:44:56.465 INFO:tasks.rados.rados.0.trial169.stdout:679: oid 34 version 99 is already newer than 99 2026-02-25T08:44:56.465 INFO:tasks.rados.rados.0.trial169.stdout:679: oid 34 version 99 is already newer than 99 2026-02-25T08:44:56.465 INFO:tasks.rados.rados.0.trial169.stdout:679: oid 34 version 99 is already newer than 99 2026-02-25T08:44:56.465 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 34 v 99 (ObjNum 322 snap 87 seq_num 322) dirty exists 2026-02-25T08:44:56.466 INFO:tasks.rados.rados.0.trial169.stdout:679: left oid 34 (ObjNum 322 snap 87 seq_num 322) 2026-02-25T08:44:56.466 INFO:tasks.rados.rados.0.trial169.stdout:684: finishing write tid 1 to trial16941936-42 2026-02-25T08:44:56.466 INFO:tasks.rados.rados.0.trial169.stdout:683: finishing write tid 1 to trial16941936-18 2026-02-25T08:44:56.466 INFO:tasks.rados.rados.0.trial169.stdout:684: finishing write tid 2 to trial16941936-42 2026-02-25T08:44:56.466 INFO:tasks.rados.rados.0.trial169.stdout:684: finishing write tid 3 to trial16941936-42 2026-02-25T08:44:56.466 INFO:tasks.rados.rados.0.trial169.stdout:686: expect (ObjNum 319 snap 86 seq_num 319) 2026-02-25T08:44:56.509 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 123643 2026-02-25T08:44:56.597 INFO:tasks.rados.rados.0.trial169.stdout:683: finishing write tid 2 to trial16941936-18 2026-02-25T08:44:56.597 INFO:tasks.rados.rados.0.trial169.stdout:684: finishing write tid 4 to trial16941936-42 2026-02-25T08:44:56.597 INFO:tasks.rados.rados.0.trial169.stdout:684: finishing write tid 5 to trial16941936-42 2026-02-25T08:44:56.597 INFO:tasks.rados.rados.0.trial169.stdout:684: oid 42 updating version 0 to 117 2026-02-25T08:44:56.598 INFO:tasks.rados.rados.0.trial169.stdout:684: oid 42 version 117 is already newer than 114 2026-02-25T08:44:56.598 INFO:tasks.rados.rados.0.trial169.stdout:684: oid 42 version 117 is already newer than 117 2026-02-25T08:44:56.598 INFO:tasks.rados.rados.0.trial169.stdout:684: oid 42 version 117 is already newer than 117 2026-02-25T08:44:56.598 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 42 v 117 (ObjNum 324 snap 87 seq_num 324) dirty exists 2026-02-25T08:44:56.598 INFO:tasks.rados.rados.0.trial169.stdout:684: left oid 42 (ObjNum 324 snap 87 seq_num 324) 2026-02-25T08:44:56.598 INFO:tasks.rados.rados.0.trial169.stdout:683: finishing write tid 3 to trial16941936-18 2026-02-25T08:44:56.598 INFO:tasks.rados.rados.0.trial169.stdout:683: finishing write tid 4 to trial16941936-18 2026-02-25T08:44:56.598 INFO:tasks.rados.rados.0.trial169.stdout:683: finishing write tid 5 to trial16941936-18 2026-02-25T08:44:56.598 INFO:tasks.rados.rados.0.trial169.stdout:683: oid 18 updating version 0 to 144 2026-02-25T08:44:56.598 INFO:tasks.rados.rados.0.trial169.stdout:683: oid 18 version 144 is already newer than 140 2026-02-25T08:44:56.598 INFO:tasks.rados.rados.0.trial169.stdout:683: oid 18 version 144 is already newer than 140 2026-02-25T08:44:56.598 INFO:tasks.rados.rados.0.trial169.stdout:683: oid 18 version 144 is already newer than 144 2026-02-25T08:44:56.598 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 18 v 144 (ObjNum 323 snap 87 seq_num 323) dirty exists 2026-02-25T08:44:56.599 INFO:tasks.rados.rados.0.trial169.stdout:683: left oid 18 (ObjNum 323 snap 87 seq_num 323) 2026-02-25T08:44:56.635 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~171s 2026-02-25T08:44:57.277 INFO:tasks.rados.rados.0.trial169.stdout:676: done (8 left) 2026-02-25T08:44:57.277 INFO:tasks.rados.rados.0.trial169.stdout:678: done (7 left) 2026-02-25T08:44:57.278 INFO:tasks.rados.rados.0.trial169.stdout:679: done (6 left) 2026-02-25T08:44:57.278 INFO:tasks.rados.rados.0.trial169.stdout:680: done (5 left) 2026-02-25T08:44:57.278 INFO:tasks.rados.rados.0.trial169.stdout:683: done (4 left) 2026-02-25T08:44:57.278 INFO:tasks.rados.rados.0.trial169.stdout:684: done (3 left) 2026-02-25T08:44:57.278 INFO:tasks.rados.rados.0.trial169.stdout:685: done (2 left) 2026-02-25T08:44:57.278 INFO:tasks.rados.rados.0.trial169.stdout:686: done (1 left) 2026-02-25T08:44:57.278 INFO:tasks.rados.rados.0.trial169.stdout:687: done (0 left) 2026-02-25T08:44:57.278 INFO:tasks.rados.rados.0.trial169.stdout:688: snap_create 2026-02-25T08:44:57.356 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:44:57.430 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:57.428+0000 7fa33e90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:57.433 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:57.431+0000 7fa33e90b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:57.436 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:57.476 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:57.477 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:44:58.145 INFO:tasks.rados.rados.0.trial169.stdout:688: done (0 left) 2026-02-25T08:44:58.146 INFO:tasks.rados.rados.0.trial169.stdout:689: write (excl) oid 44 current snap is 89 2026-02-25T08:44:58.146 INFO:tasks.rados.rados.0.trial169.stdout:689: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2026-02-25T08:44:58.150 INFO:tasks.rados.rados.0.trial169.stdout:689: writing trial16941936-44 from 662275 to 1205026 tid 1 2026-02-25T08:44:58.154 INFO:tasks.rados.rados.0.trial169.stdout:689: writing trial16941936-44 from 1785236 to 2210683 tid 2 2026-02-25T08:44:58.159 INFO:tasks.rados.rados.0.trial169.stdout:689: writing trial16941936-44 from 2942002 to 3462275 tid 3 2026-02-25T08:44:58.159 INFO:tasks.rados.rados.0.trial169.stdout:690: delete oid 38 current snap is 89 2026-02-25T08:44:58.184 INFO:tasks.rados.rados.0.trial169.stdout:689: finishing write tid 1 to trial16941936-44 2026-02-25T08:44:58.193 INFO:tasks.rados.rados.0.trial169.stdout:690: done (1 left) 2026-02-25T08:44:58.193 INFO:tasks.rados.rados.0.trial169.stdout:691: write (excl) oid 13 current snap is 89 2026-02-25T08:44:58.193 INFO:tasks.rados.rados.0.trial169.stdout:691: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2026-02-25T08:44:58.197 INFO:tasks.rados.rados.0.trial169.stdout:691: writing trial16941936-13 from 695889 to 1105491 tid 1 2026-02-25T08:44:58.201 INFO:tasks.rados.rados.0.trial169.stdout:691: writing trial16941936-13 from 1618553 to 2331316 tid 2 2026-02-25T08:44:58.206 INFO:tasks.rados.rados.0.trial169.stdout:691: writing trial16941936-13 from 2996848 to 3495889 tid 3 2026-02-25T08:44:58.206 INFO:tasks.rados.rados.0.trial169.stdout:692: delete oid 17 current snap is 89 2026-02-25T08:44:58.206 INFO:tasks.rados.rados.0.trial169.stdout:689: finishing write tid 2 to trial16941936-44 2026-02-25T08:44:58.207 INFO:tasks.rados.rados.0.trial169.stdout:689: finishing write tid 3 to trial16941936-44 2026-02-25T08:44:58.211 INFO:tasks.rados.rados.0.trial169.stdout:692: done (2 left) 2026-02-25T08:44:58.211 INFO:tasks.rados.rados.0.trial169.stdout:693: write oid 18 current snap is 89 2026-02-25T08:44:58.211 INFO:tasks.rados.rados.0.trial169.stdout:693: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2026-02-25T08:44:58.215 INFO:tasks.rados.rados.0.trial169.stdout:693: writing trial16941936-18 from 729503 to 1289603 tid 1 2026-02-25T08:44:58.221 INFO:tasks.rados.rados.0.trial169.stdout:693: writing trial16941936-18 from 2019164 to 2735596 tid 2 2026-02-25T08:44:58.224 INFO:tasks.rados.rados.0.trial169.stdout:693: writing trial16941936-18 from 3335341 to 3529503 tid 3 2026-02-25T08:44:58.224 INFO:tasks.rados.rados.0.trial169.stdout:694: read oid 42 snap -1 2026-02-25T08:44:58.225 INFO:tasks.rados.rados.0.trial169.stdout:694: expect (ObjNum 324 snap 87 seq_num 324) 2026-02-25T08:44:58.225 INFO:tasks.rados.rados.0.trial169.stdout:695: write oid 33 current snap is 89 2026-02-25T08:44:58.225 INFO:tasks.rados.rados.0.trial169.stdout:691: finishing write tid 1 to trial16941936-13 2026-02-25T08:44:58.225 INFO:tasks.rados.rados.0.trial169.stdout:695: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2026-02-25T08:44:58.229 INFO:tasks.rados.rados.0.trial169.stdout:695: writing trial16941936-33 from 746310 to 1381659 tid 1 2026-02-25T08:44:58.235 INFO:tasks.rados.rados.0.trial169.stdout:695: writing trial16941936-33 from 2077646 to 2654089 tid 2 2026-02-25T08:44:58.238 INFO:tasks.rados.rados.0.trial169.stdout:695: writing trial16941936-33 from 3362764 to 3546310 tid 3 2026-02-25T08:44:58.238 INFO:tasks.rados.rados.0.trial169.stdout:696: write (excl) oid 43 current snap is 89 2026-02-25T08:44:58.238 INFO:tasks.rados.rados.0.trial169.stdout:689: finishing write tid 4 to trial16941936-44 2026-02-25T08:44:58.239 INFO:tasks.rados.rados.0.trial169.stdout:696: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2026-02-25T08:44:58.243 INFO:tasks.rados.rados.0.trial169.stdout:696: writing trial16941936-43 from 763117 to 1473715 tid 1 2026-02-25T08:44:58.249 INFO:tasks.rados.rados.0.trial169.stdout:696: writing trial16941936-43 from 2136128 to 2856229 tid 2 2026-02-25T08:44:58.252 INFO:tasks.rados.rados.0.trial169.stdout:696: writing trial16941936-43 from 3390187 to 3563117 tid 3 2026-02-25T08:44:58.252 INFO:tasks.rados.rados.0.trial169.stdout:697: snap_create 2026-02-25T08:44:58.252 INFO:tasks.rados.rados.0.trial169.stdout:689: finishing write tid 5 to trial16941936-44 2026-02-25T08:44:58.253 INFO:tasks.rados.rados.0.trial169.stdout:689: oid 44 updating version 0 to 109 2026-02-25T08:44:58.253 INFO:tasks.rados.rados.0.trial169.stdout:689: oid 44 version 109 is already newer than 109 2026-02-25T08:44:58.253 INFO:tasks.rados.rados.0.trial169.stdout:689: oid 44 version 109 is already newer than 106 2026-02-25T08:44:58.253 INFO:tasks.rados.rados.0.trial169.stdout:689: oid 44 version 109 is already newer than 109 2026-02-25T08:44:58.253 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 44 v 109 (ObjNum 325 snap 89 seq_num 325) dirty exists 2026-02-25T08:44:58.253 INFO:tasks.rados.rados.0.trial169.stdout:689: left oid 44 (ObjNum 325 snap 89 seq_num 325) 2026-02-25T08:44:58.253 INFO:tasks.rados.rados.0.trial169.stdout:691: finishing write tid 2 to trial16941936-13 2026-02-25T08:44:58.253 INFO:tasks.rados.rados.0.trial169.stdout:691: finishing write tid 3 to trial16941936-13 2026-02-25T08:44:58.253 INFO:tasks.rados.rados.0.trial169.stdout:691: finishing write tid 4 to trial16941936-13 2026-02-25T08:44:58.253 INFO:tasks.rados.rados.0.trial169.stdout:691: finishing write tid 5 to trial16941936-13 2026-02-25T08:44:58.253 INFO:tasks.rados.rados.0.trial169.stdout:691: oid 13 updating version 0 to 294 2026-02-25T08:44:58.253 INFO:tasks.rados.rados.0.trial169.stdout:691: oid 13 version 294 is already newer than 294 2026-02-25T08:44:58.254 INFO:tasks.rados.rados.0.trial169.stdout:691: oid 13 version 294 is already newer than 294 2026-02-25T08:44:58.254 INFO:tasks.rados.rados.0.trial169.stdout:691: oid 13 version 294 is already newer than 291 2026-02-25T08:44:58.254 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 13 v 294 (ObjNum 327 snap 89 seq_num 327) dirty exists 2026-02-25T08:44:58.254 INFO:tasks.rados.rados.0.trial169.stdout:691: left oid 13 (ObjNum 327 snap 89 seq_num 327) 2026-02-25T08:44:58.254 INFO:tasks.rados.rados.0.trial169.stdout:693: finishing write tid 1 to trial16941936-18 2026-02-25T08:44:58.263 INFO:tasks.rados.rados.0.trial169.stdout:694: expect (ObjNum 324 snap 87 seq_num 324) 2026-02-25T08:44:58.313 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 3364185 2026-02-25T08:44:58.359 INFO:tasks.rados.rados.0.trial169.stdout:693: finishing write tid 2 to trial16941936-18 2026-02-25T08:44:58.359 INFO:tasks.rados.rados.0.trial169.stdout:693: finishing write tid 3 to trial16941936-18 2026-02-25T08:44:58.359 INFO:tasks.rados.rados.0.trial169.stdout:693: finishing write tid 4 to trial16941936-18 2026-02-25T08:44:58.359 INFO:tasks.rados.rados.0.trial169.stdout:693: finishing write tid 5 to trial16941936-18 2026-02-25T08:44:58.359 INFO:tasks.rados.rados.0.trial169.stdout:693: oid 18 updating version 0 to 149 2026-02-25T08:44:58.359 INFO:tasks.rados.rados.0.trial169.stdout:693: oid 18 version 149 is already newer than 149 2026-02-25T08:44:58.359 INFO:tasks.rados.rados.0.trial169.stdout:693: oid 18 version 149 is already newer than 149 2026-02-25T08:44:58.359 INFO:tasks.rados.rados.0.trial169.stdout:693: oid 18 version 149 is already newer than 149 2026-02-25T08:44:58.360 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 18 v 149 (ObjNum 329 snap 89 seq_num 329) dirty exists 2026-02-25T08:44:58.360 INFO:tasks.rados.rados.0.trial169.stdout:693: left oid 18 (ObjNum 329 snap 89 seq_num 329) 2026-02-25T08:44:58.360 INFO:tasks.rados.rados.0.trial169.stdout:695: finishing write tid 1 to trial16941936-33 2026-02-25T08:44:58.360 INFO:tasks.rados.rados.0.trial169.stdout:695: finishing write tid 2 to trial16941936-33 2026-02-25T08:44:58.360 INFO:tasks.rados.rados.0.trial169.stdout:695: finishing write tid 3 to trial16941936-33 2026-02-25T08:44:58.360 INFO:tasks.rados.rados.0.trial169.stdout:695: finishing write tid 4 to trial16941936-33 2026-02-25T08:44:58.360 INFO:tasks.rados.rados.0.trial169.stdout:696: finishing write tid 1 to trial16941936-43 2026-02-25T08:44:58.360 INFO:tasks.rados.rados.0.trial169.stdout:695: finishing write tid 5 to trial16941936-33 2026-02-25T08:44:58.360 INFO:tasks.rados.rados.0.trial169.stdout:695: oid 33 updating version 0 to 370 2026-02-25T08:44:58.360 INFO:tasks.rados.rados.0.trial169.stdout:695: oid 33 version 370 is already newer than 370 2026-02-25T08:44:58.360 INFO:tasks.rados.rados.0.trial169.stdout:695: oid 33 version 370 is already newer than 370 2026-02-25T08:44:58.360 INFO:tasks.rados.rados.0.trial169.stdout:695: oid 33 version 370 is already newer than 370 2026-02-25T08:44:58.361 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 33 v 370 (ObjNum 330 snap 89 seq_num 330) dirty exists 2026-02-25T08:44:58.361 INFO:tasks.rados.rados.0.trial169.stdout:695: left oid 33 (ObjNum 330 snap 89 seq_num 330) 2026-02-25T08:44:58.361 INFO:tasks.rados.rados.0.trial169.stdout:696: finishing write tid 2 to trial16941936-43 2026-02-25T08:44:58.361 INFO:tasks.rados.rados.0.trial169.stdout:696: finishing write tid 3 to trial16941936-43 2026-02-25T08:44:58.361 INFO:tasks.rados.rados.0.trial169.stdout:696: finishing write tid 4 to trial16941936-43 2026-02-25T08:44:58.361 INFO:tasks.rados.rados.0.trial169.stdout:696: finishing write tid 5 to trial16941936-43 2026-02-25T08:44:58.361 INFO:tasks.rados.rados.0.trial169.stdout:696: oid 43 updating version 0 to 65 2026-02-25T08:44:58.361 INFO:tasks.rados.rados.0.trial169.stdout:696: oid 43 version 65 is already newer than 65 2026-02-25T08:44:58.361 INFO:tasks.rados.rados.0.trial169.stdout:696: oid 43 version 65 is already newer than 65 2026-02-25T08:44:58.361 INFO:tasks.rados.rados.0.trial169.stdout:696: oid 43 version 65 is already newer than 62 2026-02-25T08:44:58.361 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 43 v 65 (ObjNum 331 snap 89 seq_num 331) dirty exists 2026-02-25T08:44:58.361 INFO:tasks.rados.rados.0.trial169.stdout:696: left oid 43 (ObjNum 331 snap 89 seq_num 331) 2026-02-25T08:44:59.147 INFO:tasks.rados.rados.0.trial169.stdout:689: done (6 left) 2026-02-25T08:44:59.147 INFO:tasks.rados.rados.0.trial169.stdout:691: done (5 left) 2026-02-25T08:44:59.147 INFO:tasks.rados.rados.0.trial169.stdout:693: done (4 left) 2026-02-25T08:44:59.147 INFO:tasks.rados.rados.0.trial169.stdout:694: done (3 left) 2026-02-25T08:44:59.147 INFO:tasks.rados.rados.0.trial169.stdout:695: done (2 left) 2026-02-25T08:44:59.147 INFO:tasks.rados.rados.0.trial169.stdout:696: done (1 left) 2026-02-25T08:44:59.147 INFO:tasks.rados.rados.0.trial169.stdout:697: done (0 left) 2026-02-25T08:44:59.147 INFO:tasks.rados.rados.0.trial169.stdout:698: snap_create 2026-02-25T08:44:59.478 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:44:59.555 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:59.552+0000 7fe6e630b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:59.558 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:44:59.556+0000 7fe6e630b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:44:59.561 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:44:59.603 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:44:59.604 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:00.013 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:45:00.085 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:00.087 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:00.088 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:45:00.146 INFO:tasks.rados.rados.0.trial169.stdout:698: done (0 left) 2026-02-25T08:45:00.146 INFO:tasks.rados.rados.0.trial169.stdout:699: write (excl) oid 2 current snap is 91 2026-02-25T08:45:00.146 INFO:tasks.rados.rados.0.trial169.stdout:699: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2026-02-25T08:45:00.151 INFO:tasks.rados.rados.0.trial169.stdout:699: writing trial16941936-2 from 779924 to 1565771 tid 1 2026-02-25T08:45:00.156 INFO:tasks.rados.rados.0.trial169.stdout:699: writing trial16941936-2 from 2078257 to 2658369 tid 2 2026-02-25T08:45:00.159 INFO:tasks.rados.rados.0.trial169.stdout:699: writing trial16941936-2 from 3301257 to 3579924 tid 3 2026-02-25T08:45:00.159 INFO:tasks.rados.rados.0.trial169.stdout:700: snap_create 2026-02-25T08:45:00.175 INFO:tasks.rados.rados.0.trial169.stdout:699: finishing write tid 1 to trial16941936-2 2026-02-25T08:45:00.188 INFO:tasks.rados.rados.0.trial169.stdout:699: finishing write tid 2 to trial16941936-2 2026-02-25T08:45:00.192 INFO:tasks.rados.rados.0.trial169.stdout:699: finishing write tid 3 to trial16941936-2 2026-02-25T08:45:00.195 INFO:tasks.rados.rados.0.trial169.stdout:699: finishing write tid 4 to trial16941936-2 2026-02-25T08:45:00.196 INFO:tasks.rados.rados.0.trial169.stdout:699: finishing write tid 5 to trial16941936-2 2026-02-25T08:45:00.197 INFO:tasks.rados.rados.0.trial169.stdout:699: oid 2 updating version 0 to 61 2026-02-25T08:45:00.197 INFO:tasks.rados.rados.0.trial169.stdout:699: oid 2 version 61 is already newer than 61 2026-02-25T08:45:00.197 INFO:tasks.rados.rados.0.trial169.stdout:699: oid 2 version 61 is already newer than 61 2026-02-25T08:45:00.197 INFO:tasks.rados.rados.0.trial169.stdout:699: oid 2 version 61 is already newer than 58 2026-02-25T08:45:00.197 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 2 v 61 (ObjNum 332 snap 91 seq_num 332) dirty exists 2026-02-25T08:45:00.197 INFO:tasks.rados.rados.0.trial169.stdout:699: left oid 2 (ObjNum 332 snap 91 seq_num 332) 2026-02-25T08:45:01.152 INFO:tasks.rados.rados.0.trial169.stdout:699: done (1 left) 2026-02-25T08:45:01.152 INFO:tasks.rados.rados.0.trial169.stdout:700: done (0 left) 2026-02-25T08:45:01.152 INFO:tasks.rados.rados.0.trial169.stdout:701: read oid 26 snap 89 2026-02-25T08:45:01.152 INFO:tasks.rados.rados.0.trial169.stdout:701: expect (ObjNum 314 snap 85 seq_num 314) 2026-02-25T08:45:01.152 INFO:tasks.rados.rados.0.trial169.stdout:702: snap_remove snap 88 2026-02-25T08:45:01.605 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:01.685 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:01.682+0000 7fde006b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:01.687 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:01.686+0000 7fde006b7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:01.690 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:01.731 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:02.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~176s 2026-02-25T08:45:02.155 INFO:tasks.rados.rados.0.trial169.stdout:702: done (1 left) 2026-02-25T08:45:02.155 INFO:tasks.rados.rados.0.trial169.stdout:703: read oid 8 snap -1 2026-02-25T08:45:02.155 INFO:tasks.rados.rados.0.trial169.stdout:703: expect deleted 2026-02-25T08:45:02.155 INFO:tasks.rados.rados.0.trial169.stdout:704: write (excl) oid 19 current snap is 92 2026-02-25T08:45:02.155 INFO:tasks.rados.rados.0.trial169.stdout:704: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2026-02-25T08:45:02.159 INFO:tasks.rados.rados.0.trial169.stdout:704: writing trial16941936-19 from 796731 to 1257827 tid 1 2026-02-25T08:45:02.164 INFO:tasks.rados.rados.0.trial169.stdout:704: writing trial16941936-19 from 1736739 to 2460509 tid 2 2026-02-25T08:45:02.169 INFO:tasks.rados.rados.0.trial169.stdout:704: writing trial16941936-19 from 2928680 to 3596731 tid 3 2026-02-25T08:45:02.169 INFO:tasks.rados.rados.0.trial169.stdout:705: read oid 11 snap -1 2026-02-25T08:45:02.170 INFO:tasks.rados.rados.0.trial169.stdout:705: expect (ObjNum 310 snap 84 seq_num 310) 2026-02-25T08:45:02.170 INFO:tasks.rados.rados.0.trial169.stdout:706: read oid 21 snap -1 2026-02-25T08:45:02.170 INFO:tasks.rados.rados.0.trial169.stdout:706: expect deleted 2026-02-25T08:45:02.170 INFO:tasks.rados.rados.0.trial169.stdout:707: delete oid 46 current snap is 92 2026-02-25T08:45:02.170 INFO:tasks.rados.rados.0.trial169.stdout:701: expect (ObjNum 314 snap 85 seq_num 314) 2026-02-25T08:45:02.170 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 18817 2026-02-25T08:45:02.333 INFO:tasks.rados.rados.0.trial169.stdout:704: finishing write tid 1 to trial16941936-19 2026-02-25T08:45:02.333 INFO:tasks.rados.rados.0.trial169.stdout:705: expect (ObjNum 310 snap 84 seq_num 310) 2026-02-25T08:45:02.334 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:02.374 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 304474 2026-02-25T08:45:02.401 INFO:tasks.rados.rados.0.trial169.stdout:704: finishing write tid 2 to trial16941936-19 2026-02-25T08:45:02.401 INFO:tasks.rados.rados.0.trial169.stdout:704: finishing write tid 3 to trial16941936-19 2026-02-25T08:45:02.401 INFO:tasks.rados.rados.0.trial169.stdout:704: finishing write tid 4 to trial16941936-19 2026-02-25T08:45:02.401 INFO:tasks.rados.rados.0.trial169.stdout:701: done (5 left) 2026-02-25T08:45:02.402 INFO:tasks.rados.rados.0.trial169.stdout:703: done (4 left) 2026-02-25T08:45:02.402 INFO:tasks.rados.rados.0.trial169.stdout:705: done (3 left) 2026-02-25T08:45:02.402 INFO:tasks.rados.rados.0.trial169.stdout:706: done (2 left) 2026-02-25T08:45:02.402 INFO:tasks.rados.rados.0.trial169.stdout:707: done (1 left) 2026-02-25T08:45:02.402 INFO:tasks.rados.rados.0.trial169.stdout:708: read oid 1 snap -1 2026-02-25T08:45:02.402 INFO:tasks.rados.rados.0.trial169.stdout:708: expect deleted 2026-02-25T08:45:02.402 INFO:tasks.rados.rados.0.trial169.stdout:709: delete oid 24 current snap is 92 2026-02-25T08:45:02.402 INFO:tasks.rados.rados.0.trial169.stdout:704: finishing write tid 5 to trial16941936-19 2026-02-25T08:45:02.402 INFO:tasks.rados.rados.0.trial169.stdout:704: oid 19 updating version 0 to 104 2026-02-25T08:45:02.402 INFO:tasks.rados.rados.0.trial169.stdout:704: oid 19 version 104 is already newer than 104 2026-02-25T08:45:02.402 INFO:tasks.rados.rados.0.trial169.stdout:704: oid 19 version 104 is already newer than 104 2026-02-25T08:45:02.403 INFO:tasks.rados.rados.0.trial169.stdout:704: oid 19 version 104 is already newer than 101 2026-02-25T08:45:02.403 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 19 v 104 (ObjNum 333 snap 92 seq_num 333) dirty exists 2026-02-25T08:45:02.403 INFO:tasks.rados.rados.0.trial169.stdout:704: left oid 19 (ObjNum 333 snap 92 seq_num 333) 2026-02-25T08:45:02.435 INFO:tasks.rados.rados.0.trial169.stdout:704: done (2 left) 2026-02-25T08:45:02.435 INFO:tasks.rados.rados.0.trial169.stdout:708: done (1 left) 2026-02-25T08:45:02.435 INFO:tasks.rados.rados.0.trial169.stdout:709: done (0 left) 2026-02-25T08:45:02.435 INFO:tasks.rados.rados.0.trial169.stdout:710: delete oid 46 current snap is 92 2026-02-25T08:45:02.437 INFO:tasks.rados.rados.0.trial169.stdout:710: done (0 left) 2026-02-25T08:45:02.437 INFO:tasks.rados.rados.0.trial169.stdout:711: delete oid 17 current snap is 92 2026-02-25T08:45:02.439 INFO:tasks.rados.rados.0.trial169.stdout:711: done (0 left) 2026-02-25T08:45:02.439 INFO:tasks.rados.rados.0.trial169.stdout:712: snap_create 2026-02-25T08:45:03.158 INFO:tasks.rados.rados.0.trial169.stdout:712: done (0 left) 2026-02-25T08:45:03.158 INFO:tasks.rados.rados.0.trial169.stdout:713: write (excl) oid 41 current snap is 93 2026-02-25T08:45:03.158 INFO:tasks.rados.rados.0.trial169.stdout:713: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2026-02-25T08:45:03.162 INFO:tasks.rados.rados.0.trial169.stdout:713: writing trial16941936-41 from 480766 to 1034460 tid 1 2026-02-25T08:45:03.166 INFO:tasks.rados.rados.0.trial169.stdout:713: writing trial16941936-41 from 1629149 to 2220268 tid 2 2026-02-25T08:45:03.170 INFO:tasks.rados.rados.0.trial169.stdout:713: writing trial16941936-41 from 2782148 to 3202206 tid 3 2026-02-25T08:45:03.170 INFO:tasks.rados.rados.0.trial169.stdout:714: read oid 17 snap -1 2026-02-25T08:45:03.171 INFO:tasks.rados.rados.0.trial169.stdout:714: expect deleted 2026-02-25T08:45:03.171 INFO:tasks.rados.rados.0.trial169.stdout:715: snap_create 2026-02-25T08:45:03.203 INFO:tasks.rados.rados.0.trial169.stdout:713: finishing write tid 1 to trial16941936-41 2026-02-25T08:45:03.222 INFO:tasks.rados.rados.0.trial169.stdout:713: finishing write tid 2 to trial16941936-41 2026-02-25T08:45:03.227 INFO:tasks.rados.rados.0.trial169.stdout:713: finishing write tid 3 to trial16941936-41 2026-02-25T08:45:03.231 INFO:tasks.rados.rados.0.trial169.stdout:713: finishing write tid 4 to trial16941936-41 2026-02-25T08:45:03.233 INFO:tasks.rados.rados.0.trial169.stdout:713: finishing write tid 5 to trial16941936-41 2026-02-25T08:45:03.233 INFO:tasks.rados.rados.0.trial169.stdout:713: oid 41 updating version 0 to 299 2026-02-25T08:45:03.233 INFO:tasks.rados.rados.0.trial169.stdout:713: oid 41 version 299 is already newer than 296 2026-02-25T08:45:03.233 INFO:tasks.rados.rados.0.trial169.stdout:713: oid 41 version 299 is already newer than 299 2026-02-25T08:45:03.233 INFO:tasks.rados.rados.0.trial169.stdout:713: oid 41 version 299 is already newer than 299 2026-02-25T08:45:03.233 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 41 v 299 (ObjNum 338 snap 93 seq_num 338) dirty exists 2026-02-25T08:45:03.234 INFO:tasks.rados.rados.0.trial169.stdout:713: left oid 41 (ObjNum 338 snap 93 seq_num 338) 2026-02-25T08:45:03.732 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:03.812 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:03.809+0000 7fb494ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:03.814 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:03.813+0000 7fb494ccb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:03.818 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:03.858 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:04.164 INFO:tasks.rados.rados.0.trial169.stdout:713: done (2 left) 2026-02-25T08:45:04.164 INFO:tasks.rados.rados.0.trial169.stdout:714: done (1 left) 2026-02-25T08:45:04.164 INFO:tasks.rados.rados.0.trial169.stdout:715: done (0 left) 2026-02-25T08:45:04.164 INFO:tasks.rados.rados.0.trial169.stdout:716: write oid 30 current snap is 94 2026-02-25T08:45:04.164 INFO:tasks.rados.rados.0.trial169.stdout:716: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2026-02-25T08:45:04.168 INFO:tasks.rados.rados.0.trial169.stdout:716: writing trial16941936-30 from 497573 to 1126516 tid 1 2026-02-25T08:45:04.174 INFO:tasks.rados.rados.0.trial169.stdout:716: writing trial16941936-30 from 1687631 to 2422408 tid 2 2026-02-25T08:45:04.179 INFO:tasks.rados.rados.0.trial169.stdout:716: writing trial16941936-30 from 3093218 to 3697573 tid 3 2026-02-25T08:45:04.179 INFO:tasks.rados.rados.0.trial169.stdout:717: write oid 46 current snap is 94 2026-02-25T08:45:04.179 INFO:tasks.rados.rados.0.trial169.stdout:717: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2026-02-25T08:45:04.183 INFO:tasks.rados.rados.0.trial169.stdout:717: writing trial16941936-46 from 514380 to 1218572 tid 1 2026-02-25T08:45:04.187 INFO:tasks.rados.rados.0.trial169.stdout:717: writing trial16941936-46 from 1629760 to 2108195 tid 2 2026-02-25T08:45:04.190 INFO:tasks.rados.rados.0.trial169.stdout:717: writing trial16941936-46 from 2604288 to 3046890 tid 3 2026-02-25T08:45:04.190 INFO:tasks.rados.rados.0.trial169.stdout:718: snap_remove snap 87 2026-02-25T08:45:05.089 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:45:05.162 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:05.165 INFO:tasks.rados.rados.0.trial169.stdout:718: done (2 left) 2026-02-25T08:45:05.165 INFO:tasks.rados.rados.0.trial169.stdout:719: read oid 41 snap 93 2026-02-25T08:45:05.165 INFO:tasks.rados.rados.0.trial169.stdout:719: expect (ObjNum 338 snap 93 seq_num 338) 2026-02-25T08:45:05.165 INFO:tasks.rados.rados.0.trial169.stdout:720: write (excl) oid 33 current snap is 94 2026-02-25T08:45:05.165 INFO:tasks.rados.rados.0.trial169.stdout:716: finishing write tid 1 to trial16941936-30 2026-02-25T08:45:05.165 INFO:tasks.rados.rados.0.trial169.stdout:720: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2026-02-25T08:45:05.166 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:05.167 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:45:05.170 INFO:tasks.rados.rados.0.trial169.stdout:720: writing trial16941936-33 from 531187 to 1310628 tid 1 2026-02-25T08:45:05.176 INFO:tasks.rados.rados.0.trial169.stdout:720: writing trial16941936-33 from 2088242 to 2826688 tid 2 2026-02-25T08:45:05.180 INFO:tasks.rados.rados.0.trial169.stdout:720: writing trial16941936-33 from 3431711 to 3731187 tid 3 2026-02-25T08:45:05.180 INFO:tasks.rados.rados.0.trial169.stdout:721: snap_remove snap 91 2026-02-25T08:45:05.859 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:05.940 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:05.937+0000 7fab28c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:05.942 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:05.941+0000 7fab28c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:05.946 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:05.988 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:05.988 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:06.169 INFO:tasks.rados.rados.0.trial169.stdout:721: done (4 left) 2026-02-25T08:45:06.169 INFO:tasks.rados.rados.0.trial169.stdout:722: snap_remove snap 90 2026-02-25T08:45:07.172 INFO:tasks.rados.rados.0.trial169.stdout:722: done (4 left) 2026-02-25T08:45:07.172 INFO:tasks.rados.rados.0.trial169.stdout:723: snap_create 2026-02-25T08:45:07.172 INFO:tasks.rados.rados.0.trial169.stdout:717: finishing write tid 1 to trial16941936-46 2026-02-25T08:45:07.172 INFO:tasks.rados.rados.0.trial169.stdout:716: finishing write tid 2 to trial16941936-30 2026-02-25T08:45:07.172 INFO:tasks.rados.rados.0.trial169.stdout:717: finishing write tid 2 to trial16941936-46 2026-02-25T08:45:07.172 INFO:tasks.rados.rados.0.trial169.stdout:717: finishing write tid 3 to trial16941936-46 2026-02-25T08:45:07.172 INFO:tasks.rados.rados.0.trial169.stdout:716: finishing write tid 3 to trial16941936-30 2026-02-25T08:45:07.172 INFO:tasks.rados.rados.0.trial169.stdout:716: finishing write tid 4 to trial16941936-30 2026-02-25T08:45:07.172 INFO:tasks.rados.rados.0.trial169.stdout:717: finishing write tid 4 to trial16941936-46 2026-02-25T08:45:07.173 INFO:tasks.rados.rados.0.trial169.stdout:716: finishing write tid 5 to trial16941936-30 2026-02-25T08:45:07.173 INFO:tasks.rados.rados.0.trial169.stdout:716: oid 30 updating version 0 to 113 2026-02-25T08:45:07.173 INFO:tasks.rados.rados.0.trial169.stdout:716: oid 30 version 113 is already newer than 113 2026-02-25T08:45:07.173 INFO:tasks.rados.rados.0.trial169.stdout:716: oid 30 version 113 is already newer than 113 2026-02-25T08:45:07.173 INFO:tasks.rados.rados.0.trial169.stdout:716: oid 30 version 113 is already newer than 113 2026-02-25T08:45:07.173 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 30 v 113 (ObjNum 339 snap 94 seq_num 339) dirty exists 2026-02-25T08:45:07.173 INFO:tasks.rados.rados.0.trial169.stdout:716: left oid 30 (ObjNum 339 snap 94 seq_num 339) 2026-02-25T08:45:07.173 INFO:tasks.rados.rados.0.trial169.stdout:717: finishing write tid 5 to trial16941936-46 2026-02-25T08:45:07.173 INFO:tasks.rados.rados.0.trial169.stdout:717: oid 46 updating version 0 to 121 2026-02-25T08:45:07.173 INFO:tasks.rados.rados.0.trial169.stdout:717: oid 46 version 121 is already newer than 121 2026-02-25T08:45:07.173 INFO:tasks.rados.rados.0.trial169.stdout:717: oid 46 version 121 is already newer than 121 2026-02-25T08:45:07.173 INFO:tasks.rados.rados.0.trial169.stdout:717: oid 46 version 121 is already newer than 121 2026-02-25T08:45:07.174 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 46 v 121 (ObjNum 340 snap 94 seq_num 340) dirty exists 2026-02-25T08:45:07.174 INFO:tasks.rados.rados.0.trial169.stdout:717: left oid 46 (ObjNum 340 snap 94 seq_num 340) 2026-02-25T08:45:07.174 INFO:tasks.rados.rados.0.trial169.stdout:719: expect (ObjNum 338 snap 93 seq_num 338) 2026-02-25T08:45:07.242 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2097322 2026-02-25T08:45:07.263 INFO:tasks.rados.rados.0.trial169.stdout:720: finishing write tid 1 to trial16941936-33 2026-02-25T08:45:07.264 INFO:tasks.rados.rados.0.trial169.stdout:720: finishing write tid 2 to trial16941936-33 2026-02-25T08:45:07.264 INFO:tasks.rados.rados.0.trial169.stdout:720: finishing write tid 3 to trial16941936-33 2026-02-25T08:45:07.264 INFO:tasks.rados.rados.0.trial169.stdout:720: finishing write tid 4 to trial16941936-33 2026-02-25T08:45:07.264 INFO:tasks.rados.rados.0.trial169.stdout:720: finishing write tid 5 to trial16941936-33 2026-02-25T08:45:07.264 INFO:tasks.rados.rados.0.trial169.stdout:720: oid 33 updating version 0 to 379 2026-02-25T08:45:07.264 INFO:tasks.rados.rados.0.trial169.stdout:720: oid 33 version 379 is already newer than 376 2026-02-25T08:45:07.264 INFO:tasks.rados.rados.0.trial169.stdout:720: oid 33 version 379 is already newer than 379 2026-02-25T08:45:07.264 INFO:tasks.rados.rados.0.trial169.stdout:720: oid 33 version 379 is already newer than 379 2026-02-25T08:45:07.264 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 33 v 379 (ObjNum 341 snap 94 seq_num 341) dirty exists 2026-02-25T08:45:07.264 INFO:tasks.rados.rados.0.trial169.stdout:720: left oid 33 (ObjNum 341 snap 94 seq_num 341) 2026-02-25T08:45:07.644 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~182s 2026-02-25T08:45:07.989 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:08.068 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:08.065+0000 7efe7b65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:08.070 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:08.069+0000 7efe7b65a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:08.074 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:08.115 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:08.146 INFO:tasks.rados.rados.0.trial169.stdout:716: done (4 left) 2026-02-25T08:45:08.146 INFO:tasks.rados.rados.0.trial169.stdout:717: done (3 left) 2026-02-25T08:45:08.146 INFO:tasks.rados.rados.0.trial169.stdout:719: done (2 left) 2026-02-25T08:45:08.146 INFO:tasks.rados.rados.0.trial169.stdout:720: done (1 left) 2026-02-25T08:45:08.146 INFO:tasks.rados.rados.0.trial169.stdout:723: done (0 left) 2026-02-25T08:45:08.146 INFO:tasks.rados.rados.0.trial169.stdout:724: read oid 26 snap -1 2026-02-25T08:45:08.146 INFO:tasks.rados.rados.0.trial169.stdout:724: expect (ObjNum 314 snap 85 seq_num 314) 2026-02-25T08:45:08.146 INFO:tasks.rados.rados.0.trial169.stdout:725: write oid 44 current snap is 95 2026-02-25T08:45:08.146 INFO:tasks.rados.rados.0.trial169.stdout:725: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2026-02-25T08:45:08.149 INFO:tasks.rados.rados.0.trial169.stdout:725: writing trial16941936-44 from 547994 to 1002684 tid 1 2026-02-25T08:45:08.154 INFO:tasks.rados.rados.0.trial169.stdout:725: writing trial16941936-44 from 1746724 to 2228828 tid 2 2026-02-25T08:45:08.157 INFO:tasks.rados.rados.0.trial169.stdout:725: writing trial16941936-44 from 2659134 to 3124280 tid 3 2026-02-25T08:45:08.160 INFO:tasks.rados.rados.0.trial169.stdout:725: writing trial16941936-44 from 3605976 to 3747994 tid 4 2026-02-25T08:45:08.160 INFO:tasks.rados.rados.0.trial169.stdout:726: read oid 5 snap -1 2026-02-25T08:45:08.160 INFO:tasks.rados.rados.0.trial169.stdout:726: expect (ObjNum 188 snap 53 seq_num 188) 2026-02-25T08:45:08.161 INFO:tasks.rados.rados.0.trial169.stdout:727: read oid 45 snap -1 2026-02-25T08:45:08.161 INFO:tasks.rados.rados.0.trial169.stdout:727: expect (ObjNum 173 snap 43 seq_num 173) 2026-02-25T08:45:08.161 INFO:tasks.rados.rados.0.trial169.stdout:728: write (excl) oid 35 current snap is 95 2026-02-25T08:45:08.161 INFO:tasks.rados.rados.0.trial169.stdout:728: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2026-02-25T08:45:08.164 INFO:tasks.rados.rados.0.trial169.stdout:728: writing trial16941936-35 from 564801 to 1211093 tid 1 2026-02-25T08:45:08.170 INFO:tasks.rados.rados.0.trial169.stdout:728: writing trial16941936-35 from 1921559 to 2663674 tid 2 2026-02-25T08:45:08.174 INFO:tasks.rados.rados.0.trial169.stdout:728: writing trial16941936-35 from 3202910 to 3764801 tid 3 2026-02-25T08:45:08.174 INFO:tasks.rados.rados.0.trial169.stdout:729: write oid 15 current snap is 95 2026-02-25T08:45:08.174 INFO:tasks.rados.rados.0.trial169.stdout:729: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2026-02-25T08:45:08.179 INFO:tasks.rados.rados.0.trial169.stdout:729: writing trial16941936-15 from 581608 to 1303149 tid 1 2026-02-25T08:45:08.183 INFO:tasks.rados.rados.0.trial169.stdout:729: writing trial16941936-15 from 1863688 to 2349461 tid 2 2026-02-25T08:45:08.188 INFO:tasks.rados.rados.0.trial169.stdout:729: writing trial16941936-15 from 3113980 to 3718023 tid 3 2026-02-25T08:45:08.188 INFO:tasks.rados.rados.0.trial169.stdout:730: delete oid 40 current snap is 95 2026-02-25T08:45:08.189 INFO:tasks.rados.rados.0.trial169.stdout:726: expect (ObjNum 188 snap 53 seq_num 188) 2026-02-25T08:45:08.216 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:08.270 INFO:tasks.rados.rados.0.trial169.stdout:727: expect (ObjNum 173 snap 43 seq_num 173) 2026-02-25T08:45:08.310 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 371127 2026-02-25T08:45:08.395 INFO:tasks.rados.rados.0.trial169.stdout:728: finishing write tid 1 to trial16941936-35 2026-02-25T08:45:08.395 INFO:tasks.rados.rados.0.trial169.stdout:725: finishing write tid 1 to trial16941936-44 2026-02-25T08:45:08.395 INFO:tasks.rados.rados.0.trial169.stdout:725: finishing write tid 2 to trial16941936-44 2026-02-25T08:45:08.395 INFO:tasks.rados.rados.0.trial169.stdout:726: done (6 left) 2026-02-25T08:45:08.395 INFO:tasks.rados.rados.0.trial169.stdout:727: done (5 left) 2026-02-25T08:45:08.395 INFO:tasks.rados.rados.0.trial169.stdout:730: done (4 left) 2026-02-25T08:45:08.395 INFO:tasks.rados.rados.0.trial169.stdout:731: write (excl) oid 12 current snap is 95 2026-02-25T08:45:08.395 INFO:tasks.rados.rados.0.trial169.stdout:725: finishing write tid 3 to trial16941936-44 2026-02-25T08:45:08.395 INFO:tasks.rados.rados.0.trial169.stdout:724: expect (ObjNum 314 snap 85 seq_num 314) 2026-02-25T08:45:08.396 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:08.563 INFO:tasks.rados.rados.0.trial169.stdout:728: finishing write tid 2 to trial16941936-35 2026-02-25T08:45:08.563 INFO:tasks.rados.rados.0.trial169.stdout:725: finishing write tid 4 to trial16941936-44 2026-02-25T08:45:08.563 INFO:tasks.rados.rados.0.trial169.stdout:725: finishing write tid 5 to trial16941936-44 2026-02-25T08:45:08.564 INFO:tasks.rados.rados.0.trial169.stdout:731: seq_num 346 ranges {615222=472039,1580652=489442,2768826=626587} 2026-02-25T08:45:08.567 INFO:tasks.rados.rados.0.trial169.stdout:731: writing trial16941936-12 from 615222 to 1087261 tid 1 2026-02-25T08:45:08.571 INFO:tasks.rados.rados.0.trial169.stdout:731: writing trial16941936-12 from 1580652 to 2070094 tid 2 2026-02-25T08:45:08.577 INFO:tasks.rados.rados.0.trial169.stdout:731: writing trial16941936-12 from 2768826 to 3395413 tid 3 2026-02-25T08:45:08.577 INFO:tasks.rados.rados.0.trial169.stdout:724: done (4 left) 2026-02-25T08:45:08.577 INFO:tasks.rados.rados.0.trial169.stdout:732: read oid 2 snap -1 2026-02-25T08:45:08.577 INFO:tasks.rados.rados.0.trial169.stdout:732: expect (ObjNum 332 snap 91 seq_num 332) 2026-02-25T08:45:08.577 INFO:tasks.rados.rados.0.trial169.stdout:733: write oid 43 current snap is 95 2026-02-25T08:45:08.577 INFO:tasks.rados.rados.0.trial169.stdout:725: finishing write tid 6 to trial16941936-44 2026-02-25T08:45:08.577 INFO:tasks.rados.rados.0.trial169.stdout:725: oid 44 updating version 0 to 119 2026-02-25T08:45:08.577 INFO:tasks.rados.rados.0.trial169.stdout:725: oid 44 version 119 is already newer than 119 2026-02-25T08:45:08.577 INFO:tasks.rados.rados.0.trial169.stdout:725: oid 44 version 119 is already newer than 119 2026-02-25T08:45:08.577 INFO:tasks.rados.rados.0.trial169.stdout:725: oid 44 version 119 is already newer than 119 2026-02-25T08:45:08.578 INFO:tasks.rados.rados.0.trial169.stdout:725: oid 44 version 119 is already newer than 119 2026-02-25T08:45:08.578 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 44 v 119 (ObjNum 342 snap 95 seq_num 342) dirty exists 2026-02-25T08:45:08.578 INFO:tasks.rados.rados.0.trial169.stdout:725: left oid 44 (ObjNum 342 snap 95 seq_num 342) 2026-02-25T08:45:08.578 INFO:tasks.rados.rados.0.trial169.stdout:728: finishing write tid 3 to trial16941936-35 2026-02-25T08:45:08.578 INFO:tasks.rados.rados.0.trial169.stdout:733: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2026-02-25T08:45:08.580 INFO:tasks.rados.rados.0.trial169.stdout:733: writing trial16941936-43 from 632029 to 1179317 tid 1 2026-02-25T08:45:08.585 INFO:tasks.rados.rados.0.trial169.stdout:733: writing trial16941936-43 from 1639134 to 2388587 tid 2 2026-02-25T08:45:08.588 INFO:tasks.rados.rados.0.trial169.stdout:733: writing trial16941936-43 from 2796249 to 3234108 tid 3 2026-02-25T08:45:08.591 INFO:tasks.rados.rados.0.trial169.stdout:733: writing trial16941936-43 from 3653524 to 3832029 tid 4 2026-02-25T08:45:08.591 INFO:tasks.rados.rados.0.trial169.stdout:725: done (5 left) 2026-02-25T08:45:08.592 INFO:tasks.rados.rados.0.trial169.stdout:734: delete oid 7 current snap is 95 2026-02-25T08:45:08.592 INFO:tasks.rados.rados.0.trial169.stdout:728: finishing write tid 4 to trial16941936-35 2026-02-25T08:45:08.592 INFO:tasks.rados.rados.0.trial169.stdout:728: finishing write tid 5 to trial16941936-35 2026-02-25T08:45:08.592 INFO:tasks.rados.rados.0.trial169.stdout:728: oid 35 updating version 0 to 385 2026-02-25T08:45:08.592 INFO:tasks.rados.rados.0.trial169.stdout:728: oid 35 version 385 is already newer than 385 2026-02-25T08:45:08.592 INFO:tasks.rados.rados.0.trial169.stdout:728: oid 35 version 385 is already newer than 382 2026-02-25T08:45:08.592 INFO:tasks.rados.rados.0.trial169.stdout:728: oid 35 version 385 is already newer than 385 2026-02-25T08:45:08.592 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 35 v 385 (ObjNum 343 snap 95 seq_num 343) dirty exists 2026-02-25T08:45:08.592 INFO:tasks.rados.rados.0.trial169.stdout:728: left oid 35 (ObjNum 343 snap 95 seq_num 343) 2026-02-25T08:45:08.592 INFO:tasks.rados.rados.0.trial169.stdout:729: finishing write tid 1 to trial16941936-15 2026-02-25T08:45:08.592 INFO:tasks.rados.rados.0.trial169.stdout:729: finishing write tid 2 to trial16941936-15 2026-02-25T08:45:08.593 INFO:tasks.rados.rados.0.trial169.stdout:729: finishing write tid 3 to trial16941936-15 2026-02-25T08:45:08.593 INFO:tasks.rados.rados.0.trial169.stdout:729: finishing write tid 4 to trial16941936-15 2026-02-25T08:45:08.593 INFO:tasks.rados.rados.0.trial169.stdout:729: finishing write tid 5 to trial16941936-15 2026-02-25T08:45:08.593 INFO:tasks.rados.rados.0.trial169.stdout:729: oid 15 updating version 0 to 134 2026-02-25T08:45:08.593 INFO:tasks.rados.rados.0.trial169.stdout:729: oid 15 version 134 is already newer than 134 2026-02-25T08:45:08.593 INFO:tasks.rados.rados.0.trial169.stdout:729: oid 15 version 134 is already newer than 134 2026-02-25T08:45:08.593 INFO:tasks.rados.rados.0.trial169.stdout:729: oid 15 version 134 is already newer than 134 2026-02-25T08:45:08.593 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 15 v 134 (ObjNum 344 snap 95 seq_num 344) dirty exists 2026-02-25T08:45:08.593 INFO:tasks.rados.rados.0.trial169.stdout:729: left oid 15 (ObjNum 344 snap 95 seq_num 344) 2026-02-25T08:45:08.606 INFO:tasks.rados.rados.0.trial169.stdout:731: finishing write tid 1 to trial16941936-12 2026-02-25T08:45:08.614 INFO:tasks.rados.rados.0.trial169.stdout:733: finishing write tid 1 to trial16941936-43 2026-02-25T08:45:08.621 INFO:tasks.rados.rados.0.trial169.stdout:732: expect (ObjNum 332 snap 91 seq_num 332) 2026-02-25T08:45:08.621 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:08.713 INFO:tasks.rados.rados.0.trial169.stdout:728: done (5 left) 2026-02-25T08:45:08.713 INFO:tasks.rados.rados.0.trial169.stdout:729: done (4 left) 2026-02-25T08:45:08.713 INFO:tasks.rados.rados.0.trial169.stdout:732: done (3 left) 2026-02-25T08:45:08.713 INFO:tasks.rados.rados.0.trial169.stdout:734: done (2 left) 2026-02-25T08:45:08.713 INFO:tasks.rados.rados.0.trial169.stdout:735: delete oid 44 current snap is 95 2026-02-25T08:45:08.713 INFO:tasks.rados.rados.0.trial169.stdout:733: finishing write tid 2 to trial16941936-43 2026-02-25T08:45:08.713 INFO:tasks.rados.rados.0.trial169.stdout:731: finishing write tid 2 to trial16941936-12 2026-02-25T08:45:08.713 INFO:tasks.rados.rados.0.trial169.stdout:733: finishing write tid 3 to trial16941936-43 2026-02-25T08:45:08.713 INFO:tasks.rados.rados.0.trial169.stdout:731: finishing write tid 3 to trial16941936-12 2026-02-25T08:45:08.714 INFO:tasks.rados.rados.0.trial169.stdout:731: finishing write tid 4 to trial16941936-12 2026-02-25T08:45:08.714 INFO:tasks.rados.rados.0.trial169.stdout:731: finishing write tid 5 to trial16941936-12 2026-02-25T08:45:08.714 INFO:tasks.rados.rados.0.trial169.stdout:731: oid 12 updating version 0 to 74 2026-02-25T08:45:08.714 INFO:tasks.rados.rados.0.trial169.stdout:731: oid 12 version 74 is already newer than 71 2026-02-25T08:45:08.714 INFO:tasks.rados.rados.0.trial169.stdout:731: oid 12 version 74 is already newer than 74 2026-02-25T08:45:08.714 INFO:tasks.rados.rados.0.trial169.stdout:731: oid 12 version 74 is already newer than 74 2026-02-25T08:45:08.714 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 12 v 74 (ObjNum 346 snap 95 seq_num 346) dirty exists 2026-02-25T08:45:08.714 INFO:tasks.rados.rados.0.trial169.stdout:731: left oid 12 (ObjNum 346 snap 95 seq_num 346) 2026-02-25T08:45:08.714 INFO:tasks.rados.rados.0.trial169.stdout:733: finishing write tid 4 to trial16941936-43 2026-02-25T08:45:08.714 INFO:tasks.rados.rados.0.trial169.stdout:733: finishing write tid 5 to trial16941936-43 2026-02-25T08:45:08.714 INFO:tasks.rados.rados.0.trial169.stdout:733: finishing write tid 6 to trial16941936-43 2026-02-25T08:45:08.714 INFO:tasks.rados.rados.0.trial169.stdout:733: oid 43 updating version 0 to 71 2026-02-25T08:45:08.714 INFO:tasks.rados.rados.0.trial169.stdout:733: oid 43 version 71 is already newer than 71 2026-02-25T08:45:08.715 INFO:tasks.rados.rados.0.trial169.stdout:733: oid 43 version 71 is already newer than 71 2026-02-25T08:45:08.715 INFO:tasks.rados.rados.0.trial169.stdout:733: oid 43 version 71 is already newer than 71 2026-02-25T08:45:08.715 INFO:tasks.rados.rados.0.trial169.stdout:733: oid 43 version 71 is already newer than 71 2026-02-25T08:45:08.715 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 43 v 71 (ObjNum 347 snap 95 seq_num 347) dirty exists 2026-02-25T08:45:08.715 INFO:tasks.rados.rados.0.trial169.stdout:733: left oid 43 (ObjNum 347 snap 95 seq_num 347) 2026-02-25T08:45:08.728 INFO:tasks.rados.rados.0.trial169.stdout:731: done (2 left) 2026-02-25T08:45:08.728 INFO:tasks.rados.rados.0.trial169.stdout:733: done (1 left) 2026-02-25T08:45:08.728 INFO:tasks.rados.rados.0.trial169.stdout:735: done (0 left) 2026-02-25T08:45:08.728 INFO:tasks.rados.rados.0.trial169.stdout:736: delete oid 46 current snap is 95 2026-02-25T08:45:08.751 INFO:tasks.rados.rados.0.trial169.stdout:736: done (0 left) 2026-02-25T08:45:08.751 INFO:tasks.rados.rados.0.trial169.stdout:737: delete oid 47 current snap is 95 2026-02-25T08:45:08.776 INFO:tasks.rados.rados.0.trial169.stdout:737: done (0 left) 2026-02-25T08:45:08.776 INFO:tasks.rados.rados.0.trial169.stdout:738: delete oid 44 current snap is 95 2026-02-25T08:45:08.778 INFO:tasks.rados.rados.0.trial169.stdout:738: done (0 left) 2026-02-25T08:45:08.779 INFO:tasks.rados.rados.0.trial169.stdout:739: write (excl) oid 14 current snap is 95 2026-02-25T08:45:08.779 INFO:tasks.rados.rados.0.trial169.stdout:739: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2026-02-25T08:45:08.785 INFO:tasks.rados.rados.0.trial169.stdout:739: writing trial16941936-14 from 732871 to 1448006 tid 1 2026-02-25T08:45:08.789 INFO:tasks.rados.rados.0.trial169.stdout:739: writing trial16941936-14 from 1873673 to 2517780 tid 2 2026-02-25T08:45:08.794 INFO:tasks.rados.rados.0.trial169.stdout:739: writing trial16941936-14 from 3244434 to 3932871 tid 3 2026-02-25T08:45:08.794 INFO:tasks.rados.rados.0.trial169.stdout:740: snap_create 2026-02-25T08:45:08.811 INFO:tasks.rados.rados.0.trial169.stdout:739: finishing write tid 1 to trial16941936-14 2026-02-25T08:45:08.826 INFO:tasks.rados.rados.0.trial169.stdout:739: finishing write tid 2 to trial16941936-14 2026-02-25T08:45:08.831 INFO:tasks.rados.rados.0.trial169.stdout:739: finishing write tid 3 to trial16941936-14 2026-02-25T08:45:08.833 INFO:tasks.rados.rados.0.trial169.stdout:739: finishing write tid 4 to trial16941936-14 2026-02-25T08:45:08.835 INFO:tasks.rados.rados.0.trial169.stdout:739: finishing write tid 5 to trial16941936-14 2026-02-25T08:45:08.835 INFO:tasks.rados.rados.0.trial169.stdout:739: oid 14 updating version 0 to 128 2026-02-25T08:45:08.835 INFO:tasks.rados.rados.0.trial169.stdout:739: oid 14 version 128 is already newer than 125 2026-02-25T08:45:08.835 INFO:tasks.rados.rados.0.trial169.stdout:739: oid 14 version 128 is already newer than 128 2026-02-25T08:45:08.835 INFO:tasks.rados.rados.0.trial169.stdout:739: oid 14 version 128 is already newer than 128 2026-02-25T08:45:08.835 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 14 v 128 (ObjNum 353 snap 95 seq_num 353) dirty exists 2026-02-25T08:45:08.835 INFO:tasks.rados.rados.0.trial169.stdout:739: left oid 14 (ObjNum 353 snap 95 seq_num 353) 2026-02-25T08:45:09.178 INFO:tasks.rados.rados.0.trial169.stdout:739: done (1 left) 2026-02-25T08:45:09.178 INFO:tasks.rados.rados.0.trial169.stdout:740: done (0 left) 2026-02-25T08:45:09.178 INFO:tasks.rados.rados.0.trial169.stdout:741: delete oid 1 current snap is 96 2026-02-25T08:45:09.191 INFO:tasks.rados.rados.0.trial169.stdout:741: done (0 left) 2026-02-25T08:45:09.191 INFO:tasks.rados.rados.0.trial169.stdout:742: write oid 25 current snap is 96 2026-02-25T08:45:09.191 INFO:tasks.rados.rados.0.trial169.stdout:742: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2026-02-25T08:45:09.196 INFO:tasks.rados.rados.0.trial169.stdout:742: writing trial16941936-25 from 766485 to 1232118 tid 1 2026-02-25T08:45:09.201 INFO:tasks.rados.rados.0.trial169.stdout:742: writing trial16941936-25 from 1990637 to 2638413 tid 2 2026-02-25T08:45:09.207 INFO:tasks.rados.rados.0.trial169.stdout:742: writing trial16941936-25 from 3299280 to 3966485 tid 3 2026-02-25T08:45:09.207 INFO:tasks.rados.rados.0.trial169.stdout:743: write oid 37 current snap is 96 2026-02-25T08:45:09.207 INFO:tasks.rados.rados.0.trial169.stdout:743: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2026-02-25T08:45:09.211 INFO:tasks.rados.rados.0.trial169.stdout:743: writing trial16941936-37 from 783292 to 1324174 tid 1 2026-02-25T08:45:09.215 INFO:tasks.rados.rados.0.trial169.stdout:743: writing trial16941936-37 from 1932766 to 2440553 tid 2 2026-02-25T08:45:09.221 INFO:tasks.rados.rados.0.trial169.stdout:743: writing trial16941936-37 from 3210350 to 3782363 tid 3 2026-02-25T08:45:09.221 INFO:tasks.rados.rados.0.trial169.stdout:744: snap_create 2026-02-25T08:45:09.247 INFO:tasks.rados.rados.0.trial169.stdout:742: finishing write tid 1 to trial16941936-25 2026-02-25T08:45:09.258 INFO:tasks.rados.rados.0.trial169.stdout:742: finishing write tid 2 to trial16941936-25 2026-02-25T08:45:09.265 INFO:tasks.rados.rados.0.trial169.stdout:743: finishing write tid 1 to trial16941936-37 2026-02-25T08:45:09.275 INFO:tasks.rados.rados.0.trial169.stdout:742: finishing write tid 3 to trial16941936-25 2026-02-25T08:45:09.276 INFO:tasks.rados.rados.0.trial169.stdout:743: finishing write tid 2 to trial16941936-37 2026-02-25T08:45:09.282 INFO:tasks.rados.rados.0.trial169.stdout:742: finishing write tid 4 to trial16941936-25 2026-02-25T08:45:09.283 INFO:tasks.rados.rados.0.trial169.stdout:743: finishing write tid 3 to trial16941936-37 2026-02-25T08:45:09.284 INFO:tasks.rados.rados.0.trial169.stdout:742: finishing write tid 5 to trial16941936-25 2026-02-25T08:45:09.285 INFO:tasks.rados.rados.0.trial169.stdout:742: oid 25 updating version 0 to 72 2026-02-25T08:45:09.285 INFO:tasks.rados.rados.0.trial169.stdout:742: oid 25 version 72 is already newer than 72 2026-02-25T08:45:09.285 INFO:tasks.rados.rados.0.trial169.stdout:742: oid 25 version 72 is already newer than 72 2026-02-25T08:45:09.285 INFO:tasks.rados.rados.0.trial169.stdout:742: oid 25 version 72 is already newer than 72 2026-02-25T08:45:09.285 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 25 v 72 (ObjNum 355 snap 96 seq_num 355) dirty exists 2026-02-25T08:45:09.285 INFO:tasks.rados.rados.0.trial169.stdout:742: left oid 25 (ObjNum 355 snap 96 seq_num 355) 2026-02-25T08:45:09.287 INFO:tasks.rados.rados.0.trial169.stdout:743: finishing write tid 4 to trial16941936-37 2026-02-25T08:45:09.289 INFO:tasks.rados.rados.0.trial169.stdout:743: finishing write tid 5 to trial16941936-37 2026-02-25T08:45:09.289 INFO:tasks.rados.rados.0.trial169.stdout:743: oid 37 updating version 0 to 133 2026-02-25T08:45:09.289 INFO:tasks.rados.rados.0.trial169.stdout:743: oid 37 version 133 is already newer than 133 2026-02-25T08:45:09.290 INFO:tasks.rados.rados.0.trial169.stdout:743: oid 37 version 133 is already newer than 133 2026-02-25T08:45:09.290 INFO:tasks.rados.rados.0.trial169.stdout:743: oid 37 version 133 is already newer than 133 2026-02-25T08:45:09.290 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 37 v 133 (ObjNum 356 snap 96 seq_num 356) dirty exists 2026-02-25T08:45:09.290 INFO:tasks.rados.rados.0.trial169.stdout:743: left oid 37 (ObjNum 356 snap 96 seq_num 356) 2026-02-25T08:45:10.116 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:10.168 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:45:10.180 INFO:tasks.rados.rados.0.trial169.stdout:742: done (2 left) 2026-02-25T08:45:10.180 INFO:tasks.rados.rados.0.trial169.stdout:743: done (1 left) 2026-02-25T08:45:10.180 INFO:tasks.rados.rados.0.trial169.stdout:744: done (0 left) 2026-02-25T08:45:10.180 INFO:tasks.rados.rados.0.trial169.stdout:745: write oid 23 current snap is 97 2026-02-25T08:45:10.180 INFO:tasks.rados.rados.0.trial169.stdout:745: seq_num 357 ranges {400099=616131,1591248=408851} 2026-02-25T08:45:10.184 INFO:tasks.rados.rados.0.trial169.stdout:745: writing trial16941936-23 from 400099 to 1016230 tid 1 2026-02-25T08:45:10.188 INFO:tasks.rados.rados.0.trial169.stdout:745: writing trial16941936-23 from 1591248 to 2000099 tid 2 2026-02-25T08:45:10.188 INFO:tasks.rados.rados.0.trial169.stdout:746: snap_create 2026-02-25T08:45:10.193 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:10.190+0000 7f8428d62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:10.195 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:10.194+0000 7f8428d62640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:10.199 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:10.221 INFO:tasks.rados.rados.0.trial169.stdout:745: finishing write tid 1 to trial16941936-23 2026-02-25T08:45:10.226 INFO:tasks.rados.rados.0.trial169.stdout:745: finishing write tid 2 to trial16941936-23 2026-02-25T08:45:10.233 INFO:tasks.rados.rados.0.trial169.stdout:745: finishing write tid 3 to trial16941936-23 2026-02-25T08:45:10.235 INFO:tasks.rados.rados.0.trial169.stdout:745: finishing write tid 4 to trial16941936-23 2026-02-25T08:45:10.235 INFO:tasks.rados.rados.0.trial169.stdout:745: oid 23 updating version 0 to 108 2026-02-25T08:45:10.235 INFO:tasks.rados.rados.0.trial169.stdout:745: oid 23 version 108 is already newer than 108 2026-02-25T08:45:10.235 INFO:tasks.rados.rados.0.trial169.stdout:745: oid 23 version 108 is already newer than 108 2026-02-25T08:45:10.235 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 23 v 108 (ObjNum 357 snap 97 seq_num 357) dirty exists 2026-02-25T08:45:10.236 INFO:tasks.rados.rados.0.trial169.stdout:745: left oid 23 (ObjNum 357 snap 97 seq_num 357) 2026-02-25T08:45:10.239 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:10.242 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:10.242 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:45:10.261 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:10.261 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:11.182 INFO:tasks.rados.rados.0.trial169.stdout:745: done (1 left) 2026-02-25T08:45:11.183 INFO:tasks.rados.rados.0.trial169.stdout:746: done (0 left) 2026-02-25T08:45:11.183 INFO:tasks.rados.rados.0.trial169.stdout:747: read oid 17 snap -1 2026-02-25T08:45:11.183 INFO:tasks.rados.rados.0.trial169.stdout:747: expect deleted 2026-02-25T08:45:11.183 INFO:tasks.rados.rados.0.trial169.stdout:748: read oid 50 snap -1 2026-02-25T08:45:11.183 INFO:tasks.rados.rados.0.trial169.stdout:748: expect (ObjNum 253 snap 72 seq_num 253) 2026-02-25T08:45:11.183 INFO:tasks.rados.rados.0.trial169.stdout:749: snap_remove snap 94 2026-02-25T08:45:12.185 INFO:tasks.rados.rados.0.trial169.stdout:749: done (2 left) 2026-02-25T08:45:12.185 INFO:tasks.rados.rados.0.trial169.stdout:750: write oid 19 current snap is 98 2026-02-25T08:45:12.185 INFO:tasks.rados.rados.0.trial169.stdout:750: seq_num 358 ranges {416906=407733,1366083=511456} 2026-02-25T08:45:12.188 INFO:tasks.rados.rados.0.trial169.stdout:750: writing trial16941936-19 from 416906 to 824639 tid 1 2026-02-25T08:45:12.192 INFO:tasks.rados.rados.0.trial169.stdout:750: writing trial16941936-19 from 1366083 to 1877539 tid 2 2026-02-25T08:45:12.193 INFO:tasks.rados.rados.0.trial169.stdout:751: snap_remove snap 97 2026-02-25T08:45:12.262 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:12.337 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:12.334+0000 7fde0370b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:12.339 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:12.337+0000 7fde0370b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:12.343 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:12.383 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:12.384 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:13.149 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~187s 2026-02-25T08:45:13.190 INFO:tasks.rados.rados.0.trial169.stdout:751: done (3 left) 2026-02-25T08:45:13.190 INFO:tasks.rados.rados.0.trial169.stdout:752: read oid 33 snap -1 2026-02-25T08:45:13.190 INFO:tasks.rados.rados.0.trial169.stdout:752: expect (ObjNum 341 snap 94 seq_num 341) 2026-02-25T08:45:13.190 INFO:tasks.rados.rados.0.trial169.stdout:753: read oid 9 snap -1 2026-02-25T08:45:13.190 INFO:tasks.rados.rados.0.trial169.stdout:753: expect (ObjNum 270 snap 75 seq_num 270) 2026-02-25T08:45:13.190 INFO:tasks.rados.rados.0.trial169.stdout:754: write oid 43 current snap is 98 2026-02-25T08:45:13.190 INFO:tasks.rados.rados.0.trial169.stdout:754: seq_num 359 ranges {433713=482982,1424565=609148} 2026-02-25T08:45:13.193 INFO:tasks.rados.rados.0.trial169.stdout:754: writing trial16941936-43 from 433713 to 916695 tid 1 2026-02-25T08:45:13.197 INFO:tasks.rados.rados.0.trial169.stdout:754: writing trial16941936-43 from 1424565 to 2033713 tid 2 2026-02-25T08:45:13.197 INFO:tasks.rados.rados.0.trial169.stdout:755: delete oid 13 current snap is 98 2026-02-25T08:45:13.197 INFO:tasks.rados.rados.0.trial169.stdout:748: expect (ObjNum 253 snap 72 seq_num 253) 2026-02-25T08:45:13.197 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:13.302 INFO:tasks.rados.rados.0.trial169.stdout:750: finishing write tid 1 to trial16941936-19 2026-02-25T08:45:13.303 INFO:tasks.rados.rados.0.trial169.stdout:750: finishing write tid 2 to trial16941936-19 2026-02-25T08:45:13.303 INFO:tasks.rados.rados.0.trial169.stdout:747: done (6 left) 2026-02-25T08:45:13.303 INFO:tasks.rados.rados.0.trial169.stdout:748: done (5 left) 2026-02-25T08:45:13.303 INFO:tasks.rados.rados.0.trial169.stdout:755: done (4 left) 2026-02-25T08:45:13.303 INFO:tasks.rados.rados.0.trial169.stdout:756: write oid 26 current snap is 98 2026-02-25T08:45:13.303 INFO:tasks.rados.rados.0.trial169.stdout:750: finishing write tid 3 to trial16941936-19 2026-02-25T08:45:13.303 INFO:tasks.rados.rados.0.trial169.stdout:756: seq_num 361 ranges {467327=633480,1825176=242151} 2026-02-25T08:45:13.307 INFO:tasks.rados.rados.0.trial169.stdout:756: writing trial16941936-26 from 467327 to 1100807 tid 1 2026-02-25T08:45:13.311 INFO:tasks.rados.rados.0.trial169.stdout:756: writing trial16941936-26 from 1825176 to 2067327 tid 2 2026-02-25T08:45:13.311 INFO:tasks.rados.rados.0.trial169.stdout:757: write (excl) oid 1 current snap is 98 2026-02-25T08:45:13.311 INFO:tasks.rados.rados.0.trial169.stdout:750: finishing write tid 4 to trial16941936-19 2026-02-25T08:45:13.311 INFO:tasks.rados.rados.0.trial169.stdout:750: oid 19 updating version 0 to 112 2026-02-25T08:45:13.311 INFO:tasks.rados.rados.0.trial169.stdout:750: oid 19 version 112 is already newer than 112 2026-02-25T08:45:13.311 INFO:tasks.rados.rados.0.trial169.stdout:750: oid 19 version 112 is already newer than 112 2026-02-25T08:45:13.311 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 19 v 112 (ObjNum 358 snap 98 seq_num 358) dirty exists 2026-02-25T08:45:13.311 INFO:tasks.rados.rados.0.trial169.stdout:750: left oid 19 (ObjNum 358 snap 98 seq_num 358) 2026-02-25T08:45:13.311 INFO:tasks.rados.rados.0.trial169.stdout:753: expect (ObjNum 270 snap 75 seq_num 270) 2026-02-25T08:45:13.335 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:13.356 INFO:tasks.rados.rados.0.trial169.stdout:752: expect (ObjNum 341 snap 94 seq_num 341) 2026-02-25T08:45:13.356 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:13.418 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1659960 2026-02-25T08:45:13.458 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:13.513 INFO:tasks.rados.rados.0.trial169.stdout:754: finishing write tid 1 to trial16941936-43 2026-02-25T08:45:13.513 INFO:tasks.rados.rados.0.trial169.stdout:754: finishing write tid 2 to trial16941936-43 2026-02-25T08:45:13.513 INFO:tasks.rados.rados.0.trial169.stdout:757: seq_num 362 ranges {484134=708729,1883658=200476} 2026-02-25T08:45:13.518 INFO:tasks.rados.rados.0.trial169.stdout:757: writing trial16941936-1 from 484134 to 1192863 tid 1 2026-02-25T08:45:13.522 INFO:tasks.rados.rados.0.trial169.stdout:757: writing trial16941936-1 from 1883658 to 2084134 tid 2 2026-02-25T08:45:13.522 INFO:tasks.rados.rados.0.trial169.stdout:750: done (5 left) 2026-02-25T08:45:13.522 INFO:tasks.rados.rados.0.trial169.stdout:752: done (4 left) 2026-02-25T08:45:13.522 INFO:tasks.rados.rados.0.trial169.stdout:753: done (3 left) 2026-02-25T08:45:13.523 INFO:tasks.rados.rados.0.trial169.stdout:758: read oid 22 snap 89 2026-02-25T08:45:13.523 INFO:tasks.rados.rados.0.trial169.stdout:758: expect (ObjNum 282 snap 76 seq_num 282) 2026-02-25T08:45:13.523 INFO:tasks.rados.rados.0.trial169.stdout:759: write oid 39 current snap is 98 2026-02-25T08:45:13.523 INFO:tasks.rados.rados.0.trial169.stdout:754: finishing write tid 3 to trial16941936-43 2026-02-25T08:45:13.523 INFO:tasks.rados.rados.0.trial169.stdout:754: finishing write tid 4 to trial16941936-43 2026-02-25T08:45:13.523 INFO:tasks.rados.rados.0.trial169.stdout:754: oid 43 updating version 0 to 75 2026-02-25T08:45:13.523 INFO:tasks.rados.rados.0.trial169.stdout:754: oid 43 version 75 is already newer than 75 2026-02-25T08:45:13.523 INFO:tasks.rados.rados.0.trial169.stdout:754: oid 43 version 75 is already newer than 75 2026-02-25T08:45:13.523 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 43 v 75 (ObjNum 359 snap 98 seq_num 359) dirty exists 2026-02-25T08:45:13.523 INFO:tasks.rados.rados.0.trial169.stdout:754: left oid 43 (ObjNum 359 snap 98 seq_num 359) 2026-02-25T08:45:13.523 INFO:tasks.rados.rados.0.trial169.stdout:756: finishing write tid 1 to trial16941936-26 2026-02-25T08:45:13.523 INFO:tasks.rados.rados.0.trial169.stdout:756: finishing write tid 2 to trial16941936-26 2026-02-25T08:45:13.524 INFO:tasks.rados.rados.0.trial169.stdout:756: finishing write tid 3 to trial16941936-26 2026-02-25T08:45:13.524 INFO:tasks.rados.rados.0.trial169.stdout:756: finishing write tid 4 to trial16941936-26 2026-02-25T08:45:13.524 INFO:tasks.rados.rados.0.trial169.stdout:756: oid 26 updating version 0 to 309 2026-02-25T08:45:13.524 INFO:tasks.rados.rados.0.trial169.stdout:756: oid 26 version 309 is already newer than 309 2026-02-25T08:45:13.524 INFO:tasks.rados.rados.0.trial169.stdout:756: oid 26 version 309 is already newer than 309 2026-02-25T08:45:13.524 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 26 v 309 (ObjNum 361 snap 98 seq_num 361) dirty exists 2026-02-25T08:45:13.524 INFO:tasks.rados.rados.0.trial169.stdout:756: left oid 26 (ObjNum 361 snap 98 seq_num 361) 2026-02-25T08:45:13.524 INFO:tasks.rados.rados.0.trial169.stdout:759: seq_num 363 ranges {500941=783978,1942140=158801} 2026-02-25T08:45:13.526 INFO:tasks.rados.rados.0.trial169.stdout:759: writing trial16941936-39 from 500941 to 1284919 tid 1 2026-02-25T08:45:13.529 INFO:tasks.rados.rados.0.trial169.stdout:759: writing trial16941936-39 from 1942140 to 2100941 tid 2 2026-02-25T08:45:13.529 INFO:tasks.rados.rados.0.trial169.stdout:754: done (4 left) 2026-02-25T08:45:13.529 INFO:tasks.rados.rados.0.trial169.stdout:756: done (3 left) 2026-02-25T08:45:13.530 INFO:tasks.rados.rados.0.trial169.stdout:760: write (excl) oid 5 current snap is 98 2026-02-25T08:45:13.530 INFO:tasks.rados.rados.0.trial169.stdout:760: seq_num 364 ranges {517748=459227,1600622=406110} 2026-02-25T08:45:13.533 INFO:tasks.rados.rados.0.trial169.stdout:760: writing trial16941936-5 from 517748 to 976975 tid 1 2026-02-25T08:45:13.537 INFO:tasks.rados.rados.0.trial169.stdout:760: writing trial16941936-5 from 1600622 to 2006732 tid 2 2026-02-25T08:45:13.537 INFO:tasks.rados.rados.0.trial169.stdout:761: write (excl) oid 33 current snap is 98 2026-02-25T08:45:13.537 INFO:tasks.rados.rados.0.trial169.stdout:761: seq_num 365 ranges {534555=650829,1659104=475451} 2026-02-25T08:45:13.541 INFO:tasks.rados.rados.0.trial169.stdout:761: writing trial16941936-33 from 534555 to 1185384 tid 1 2026-02-25T08:45:13.546 INFO:tasks.rados.rados.0.trial169.stdout:761: writing trial16941936-33 from 1659104 to 2134555 tid 2 2026-02-25T08:45:13.546 INFO:tasks.rados.rados.0.trial169.stdout:762: read oid 19 snap -1 2026-02-25T08:45:13.546 INFO:tasks.rados.rados.0.trial169.stdout:762: expect (ObjNum 358 snap 98 seq_num 358) 2026-02-25T08:45:13.546 INFO:tasks.rados.rados.0.trial169.stdout:763: read oid 11 snap -1 2026-02-25T08:45:13.546 INFO:tasks.rados.rados.0.trial169.stdout:763: expect (ObjNum 310 snap 84 seq_num 310) 2026-02-25T08:45:13.546 INFO:tasks.rados.rados.0.trial169.stdout:764: write oid 50 current snap is 98 2026-02-25T08:45:13.546 INFO:tasks.rados.rados.0.trial169.stdout:758: expect (ObjNum 282 snap 76 seq_num 282) 2026-02-25T08:45:13.601 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2649687 2026-02-25T08:45:13.602 INFO:tasks.rados.rados.0.trial169.stdout:757: finishing write tid 1 to trial16941936-1 2026-02-25T08:45:13.602 INFO:tasks.rados.rados.0.trial169.stdout:764: seq_num 366 ranges {551362=726078,1717586=409779} 2026-02-25T08:45:13.607 INFO:tasks.rados.rados.0.trial169.stdout:764: writing trial16941936-50 from 551362 to 1277440 tid 1 2026-02-25T08:45:13.610 INFO:tasks.rados.rados.0.trial169.stdout:764: writing trial16941936-50 from 1717586 to 2127365 tid 2 2026-02-25T08:45:13.610 INFO:tasks.rados.rados.0.trial169.stdout:758: done (7 left) 2026-02-25T08:45:13.610 INFO:tasks.rados.rados.0.trial169.stdout:765: write (excl) oid 13 current snap is 98 2026-02-25T08:45:13.611 INFO:tasks.rados.rados.0.trial169.stdout:757: finishing write tid 2 to trial16941936-1 2026-02-25T08:45:13.611 INFO:tasks.rados.rados.0.trial169.stdout:765: seq_num 367 ranges {568169=401327,1376068=669790} 2026-02-25T08:45:13.613 INFO:tasks.rados.rados.0.trial169.stdout:765: writing trial16941936-13 from 568169 to 969496 tid 1 2026-02-25T08:45:13.618 INFO:tasks.rados.rados.0.trial169.stdout:765: writing trial16941936-13 from 1376068 to 2045858 tid 2 2026-02-25T08:45:13.618 INFO:tasks.rados.rados.0.trial169.stdout:766: read oid 15 snap -1 2026-02-25T08:45:13.619 INFO:tasks.rados.rados.0.trial169.stdout:766: expect (ObjNum 344 snap 95 seq_num 344) 2026-02-25T08:45:13.619 INFO:tasks.rados.rados.0.trial169.stdout:759: finishing write tid 1 to trial16941936-39 2026-02-25T08:45:13.619 INFO:tasks.rados.rados.0.trial169.stdout:759: finishing write tid 2 to trial16941936-39 2026-02-25T08:45:13.619 INFO:tasks.rados.rados.0.trial169.stdout:767: read oid 6 snap -1 2026-02-25T08:45:13.619 INFO:tasks.rados.rados.0.trial169.stdout:767: expect (ObjNum 288 snap 79 seq_num 288) 2026-02-25T08:45:13.619 INFO:tasks.rados.rados.0.trial169.stdout:768: write (excl) oid 34 current snap is 98 2026-02-25T08:45:13.619 INFO:tasks.rados.rados.0.trial169.stdout:768: seq_num 368 ranges {584976=476576,1834550=350426} 2026-02-25T08:45:13.622 INFO:tasks.rados.rados.0.trial169.stdout:768: writing trial16941936-34 from 584976 to 1061552 tid 1 2026-02-25T08:45:13.626 INFO:tasks.rados.rados.0.trial169.stdout:768: writing trial16941936-34 from 1834550 to 2184976 tid 2 2026-02-25T08:45:13.626 INFO:tasks.rados.rados.0.trial169.stdout:769: snap_create 2026-02-25T08:45:13.627 INFO:tasks.rados.rados.0.trial169.stdout:757: finishing write tid 3 to trial16941936-1 2026-02-25T08:45:13.627 INFO:tasks.rados.rados.0.trial169.stdout:759: finishing write tid 3 to trial16941936-39 2026-02-25T08:45:13.627 INFO:tasks.rados.rados.0.trial169.stdout:757: finishing write tid 4 to trial16941936-1 2026-02-25T08:45:13.627 INFO:tasks.rados.rados.0.trial169.stdout:757: oid 1 updating version 0 to 75 2026-02-25T08:45:13.627 INFO:tasks.rados.rados.0.trial169.stdout:757: oid 1 version 75 is already newer than 75 2026-02-25T08:45:13.627 INFO:tasks.rados.rados.0.trial169.stdout:757: oid 1 version 75 is already newer than 73 2026-02-25T08:45:13.627 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 1 v 75 (ObjNum 362 snap 98 seq_num 362) dirty exists 2026-02-25T08:45:13.627 INFO:tasks.rados.rados.0.trial169.stdout:757: left oid 1 (ObjNum 362 snap 98 seq_num 362) 2026-02-25T08:45:13.627 INFO:tasks.rados.rados.0.trial169.stdout:759: finishing write tid 4 to trial16941936-39 2026-02-25T08:45:13.627 INFO:tasks.rados.rados.0.trial169.stdout:759: oid 39 updating version 0 to 123 2026-02-25T08:45:13.627 INFO:tasks.rados.rados.0.trial169.stdout:759: oid 39 version 123 is already newer than 123 2026-02-25T08:45:13.628 INFO:tasks.rados.rados.0.trial169.stdout:759: oid 39 version 123 is already newer than 123 2026-02-25T08:45:13.628 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 39 v 123 (ObjNum 363 snap 98 seq_num 363) dirty exists 2026-02-25T08:45:13.628 INFO:tasks.rados.rados.0.trial169.stdout:759: left oid 39 (ObjNum 363 snap 98 seq_num 363) 2026-02-25T08:45:13.628 INFO:tasks.rados.rados.0.trial169.stdout:762: expect (ObjNum 358 snap 98 seq_num 358) 2026-02-25T08:45:13.628 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:13.684 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1856314 2026-02-25T08:45:13.690 INFO:tasks.rados.rados.0.trial169.stdout:763: expect (ObjNum 310 snap 84 seq_num 310) 2026-02-25T08:45:13.690 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:13.716 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:13.743 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:13.771 INFO:tasks.rados.rados.0.trial169.stdout:760: finishing write tid 1 to trial16941936-5 2026-02-25T08:45:13.771 INFO:tasks.rados.rados.0.trial169.stdout:760: finishing write tid 2 to trial16941936-5 2026-02-25T08:45:13.771 INFO:tasks.rados.rados.0.trial169.stdout:760: finishing write tid 3 to trial16941936-5 2026-02-25T08:45:13.771 INFO:tasks.rados.rados.0.trial169.stdout:760: finishing write tid 4 to trial16941936-5 2026-02-25T08:45:13.771 INFO:tasks.rados.rados.0.trial169.stdout:760: oid 5 updating version 0 to 155 2026-02-25T08:45:13.772 INFO:tasks.rados.rados.0.trial169.stdout:760: oid 5 updating version 155 to 157 2026-02-25T08:45:13.772 INFO:tasks.rados.rados.0.trial169.stdout:760: oid 5 version 157 is already newer than 157 2026-02-25T08:45:13.772 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 5 v 157 (ObjNum 364 snap 98 seq_num 364) dirty exists 2026-02-25T08:45:13.772 INFO:tasks.rados.rados.0.trial169.stdout:760: left oid 5 (ObjNum 364 snap 98 seq_num 364) 2026-02-25T08:45:13.772 INFO:tasks.rados.rados.0.trial169.stdout:761: finishing write tid 1 to trial16941936-33 2026-02-25T08:45:13.772 INFO:tasks.rados.rados.0.trial169.stdout:761: finishing write tid 2 to trial16941936-33 2026-02-25T08:45:13.772 INFO:tasks.rados.rados.0.trial169.stdout:765: finishing write tid 1 to trial16941936-13 2026-02-25T08:45:13.772 INFO:tasks.rados.rados.0.trial169.stdout:767: expect (ObjNum 288 snap 79 seq_num 288) 2026-02-25T08:45:13.810 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:13.835 INFO:tasks.rados.rados.0.trial169.stdout:761: finishing write tid 3 to trial16941936-33 2026-02-25T08:45:13.835 INFO:tasks.rados.rados.0.trial169.stdout:765: finishing write tid 2 to trial16941936-13 2026-02-25T08:45:13.835 INFO:tasks.rados.rados.0.trial169.stdout:761: finishing write tid 4 to trial16941936-33 2026-02-25T08:45:13.835 INFO:tasks.rados.rados.0.trial169.stdout:761: oid 33 updating version 0 to 397 2026-02-25T08:45:13.835 INFO:tasks.rados.rados.0.trial169.stdout:761: oid 33 version 397 is already newer than 397 2026-02-25T08:45:13.835 INFO:tasks.rados.rados.0.trial169.stdout:761: oid 33 version 397 is already newer than 395 2026-02-25T08:45:13.835 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 33 v 397 (ObjNum 365 snap 98 seq_num 365) dirty exists 2026-02-25T08:45:13.835 INFO:tasks.rados.rados.0.trial169.stdout:761: left oid 33 (ObjNum 365 snap 98 seq_num 365) 2026-02-25T08:45:13.835 INFO:tasks.rados.rados.0.trial169.stdout:766: expect (ObjNum 344 snap 95 seq_num 344) 2026-02-25T08:45:13.871 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:13.921 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1256927 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:765: finishing write tid 3 to trial16941936-13 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:765: finishing write tid 4 to trial16941936-13 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:765: oid 13 updating version 0 to 310 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:765: oid 13 updating version 310 to 312 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:765: oid 13 version 312 is already newer than 312 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 13 v 312 (ObjNum 367 snap 98 seq_num 367) dirty exists 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:765: left oid 13 (ObjNum 367 snap 98 seq_num 367) 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:768: finishing write tid 1 to trial16941936-34 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:768: finishing write tid 2 to trial16941936-34 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:764: finishing write tid 1 to trial16941936-50 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:764: finishing write tid 2 to trial16941936-50 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:764: finishing write tid 3 to trial16941936-50 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:768: finishing write tid 3 to trial16941936-34 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:764: finishing write tid 4 to trial16941936-50 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:764: oid 50 updating version 0 to 401 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:764: oid 50 version 401 is already newer than 401 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:764: oid 50 version 401 is already newer than 401 2026-02-25T08:45:13.954 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 50 v 401 (ObjNum 366 snap 98 seq_num 366) dirty exists 2026-02-25T08:45:13.955 INFO:tasks.rados.rados.0.trial169.stdout:764: left oid 50 (ObjNum 366 snap 98 seq_num 366) 2026-02-25T08:45:13.955 INFO:tasks.rados.rados.0.trial169.stdout:768: finishing write tid 4 to trial16941936-34 2026-02-25T08:45:13.955 INFO:tasks.rados.rados.0.trial169.stdout:768: oid 34 updating version 0 to 114 2026-02-25T08:45:13.955 INFO:tasks.rados.rados.0.trial169.stdout:768: oid 34 updating version 114 to 116 2026-02-25T08:45:13.955 INFO:tasks.rados.rados.0.trial169.stdout:768: oid 34 version 116 is already newer than 116 2026-02-25T08:45:13.955 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 34 v 116 (ObjNum 368 snap 98 seq_num 368) dirty exists 2026-02-25T08:45:13.955 INFO:tasks.rados.rados.0.trial169.stdout:768: left oid 34 (ObjNum 368 snap 98 seq_num 368) 2026-02-25T08:45:14.196 INFO:tasks.rados.rados.0.trial169.stdout:757: done (11 left) 2026-02-25T08:45:14.196 INFO:tasks.rados.rados.0.trial169.stdout:759: done (10 left) 2026-02-25T08:45:14.196 INFO:tasks.rados.rados.0.trial169.stdout:760: done (9 left) 2026-02-25T08:45:14.196 INFO:tasks.rados.rados.0.trial169.stdout:761: done (8 left) 2026-02-25T08:45:14.196 INFO:tasks.rados.rados.0.trial169.stdout:762: done (7 left) 2026-02-25T08:45:14.196 INFO:tasks.rados.rados.0.trial169.stdout:763: done (6 left) 2026-02-25T08:45:14.196 INFO:tasks.rados.rados.0.trial169.stdout:764: done (5 left) 2026-02-25T08:45:14.196 INFO:tasks.rados.rados.0.trial169.stdout:765: done (4 left) 2026-02-25T08:45:14.197 INFO:tasks.rados.rados.0.trial169.stdout:766: done (3 left) 2026-02-25T08:45:14.197 INFO:tasks.rados.rados.0.trial169.stdout:767: done (2 left) 2026-02-25T08:45:14.197 INFO:tasks.rados.rados.0.trial169.stdout:768: done (1 left) 2026-02-25T08:45:14.197 INFO:tasks.rados.rados.0.trial169.stdout:769: done (0 left) 2026-02-25T08:45:14.197 INFO:tasks.rados.rados.0.trial169.stdout:770: snap_create 2026-02-25T08:45:14.385 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:14.463 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:14.460+0000 7fe1d285a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:14.466 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:14.464+0000 7fe1d285a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:14.469 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:14.509 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:14.509 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:15.198 INFO:tasks.rados.rados.0.trial169.stdout:770: done (0 left) 2026-02-25T08:45:15.198 INFO:tasks.rados.rados.0.trial169.stdout:771: read oid 24 snap 93 2026-02-25T08:45:15.198 INFO:tasks.rados.rados.0.trial169.stdout:771: expect deleted 2026-02-25T08:45:15.198 INFO:tasks.rados.rados.0.trial169.stdout:772: write (excl) oid 50 current snap is 100 2026-02-25T08:45:15.198 INFO:tasks.rados.rados.0.trial169.stdout:772: seq_num 369 ranges {601783=551825,1776679=425104} 2026-02-25T08:45:15.203 INFO:tasks.rados.rados.0.trial169.stdout:772: writing trial16941936-50 from 601783 to 1153608 tid 1 2026-02-25T08:45:15.207 INFO:tasks.rados.rados.0.trial169.stdout:772: writing trial16941936-50 from 1776679 to 2201783 tid 2 2026-02-25T08:45:15.207 INFO:tasks.rados.rados.0.trial169.stdout:773: read oid 36 snap -1 2026-02-25T08:45:15.207 INFO:tasks.rados.rados.0.trial169.stdout:773: expect deleted 2026-02-25T08:45:15.207 INFO:tasks.rados.rados.0.trial169.stdout:774: write (excl) oid 28 current snap is 100 2026-02-25T08:45:15.207 INFO:tasks.rados.rados.0.trial169.stdout:774: seq_num 370 ranges {618590=627074,1835161=383429} 2026-02-25T08:45:15.211 INFO:tasks.rados.rados.0.trial169.stdout:774: writing trial16941936-28 from 618590 to 1245664 tid 1 2026-02-25T08:45:15.215 INFO:tasks.rados.rados.0.trial169.stdout:774: writing trial16941936-28 from 1835161 to 2218590 tid 2 2026-02-25T08:45:15.215 INFO:tasks.rados.rados.0.trial169.stdout:775: read oid 49 snap -1 2026-02-25T08:45:15.215 INFO:tasks.rados.rados.0.trial169.stdout:775: expect (ObjNum 295 snap 81 seq_num 295) 2026-02-25T08:45:15.215 INFO:tasks.rados.rados.0.trial169.stdout:776: write oid 47 current snap is 100 2026-02-25T08:45:15.216 INFO:tasks.rados.rados.0.trial169.stdout:776: seq_num 371 ranges {635397=702323,1893643=341754} 2026-02-25T08:45:15.220 INFO:tasks.rados.rados.0.trial169.stdout:776: writing trial16941936-47 from 635397 to 1337720 tid 1 2026-02-25T08:45:15.223 INFO:tasks.rados.rados.0.trial169.stdout:776: writing trial16941936-47 from 1893643 to 2235397 tid 2 2026-02-25T08:45:15.223 INFO:tasks.rados.rados.0.trial169.stdout:777: snap_remove snap 89 2026-02-25T08:45:15.244 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:45:15.314 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:15.318 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:45:16.204 INFO:tasks.rados.rados.0.trial169.stdout:777: done (6 left) 2026-02-25T08:45:16.204 INFO:tasks.rados.rados.0.trial169.stdout:778: snap_create 2026-02-25T08:45:16.204 INFO:tasks.rados.rados.0.trial169.stdout:775: expect (ObjNum 295 snap 81 seq_num 295) 2026-02-25T08:45:16.309 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2271490 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:776: finishing write tid 1 to trial16941936-47 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:774: finishing write tid 1 to trial16941936-28 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:776: finishing write tid 2 to trial16941936-47 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:774: finishing write tid 2 to trial16941936-28 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:776: finishing write tid 3 to trial16941936-47 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:776: finishing write tid 4 to trial16941936-47 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:776: oid 47 updating version 0 to 78 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:776: oid 47 version 78 is already newer than 78 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:776: oid 47 version 78 is already newer than 78 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 47 v 78 (ObjNum 371 snap 100 seq_num 371) dirty exists 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:776: left oid 47 (ObjNum 371 snap 100 seq_num 371) 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:774: finishing write tid 3 to trial16941936-28 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:774: finishing write tid 4 to trial16941936-28 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:774: oid 28 updating version 0 to 53 2026-02-25T08:45:16.317 INFO:tasks.rados.rados.0.trial169.stdout:774: oid 28 version 53 is already newer than 53 2026-02-25T08:45:16.318 INFO:tasks.rados.rados.0.trial169.stdout:774: oid 28 version 53 is already newer than 51 2026-02-25T08:45:16.318 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 28 v 53 (ObjNum 370 snap 100 seq_num 370) dirty exists 2026-02-25T08:45:16.318 INFO:tasks.rados.rados.0.trial169.stdout:774: left oid 28 (ObjNum 370 snap 100 seq_num 370) 2026-02-25T08:45:16.318 INFO:tasks.rados.rados.0.trial169.stdout:772: finishing write tid 1 to trial16941936-50 2026-02-25T08:45:16.318 INFO:tasks.rados.rados.0.trial169.stdout:772: finishing write tid 2 to trial16941936-50 2026-02-25T08:45:16.318 INFO:tasks.rados.rados.0.trial169.stdout:772: finishing write tid 3 to trial16941936-50 2026-02-25T08:45:16.318 INFO:tasks.rados.rados.0.trial169.stdout:772: finishing write tid 4 to trial16941936-50 2026-02-25T08:45:16.318 INFO:tasks.rados.rados.0.trial169.stdout:772: oid 50 updating version 0 to 405 2026-02-25T08:45:16.318 INFO:tasks.rados.rados.0.trial169.stdout:772: oid 50 version 405 is already newer than 403 2026-02-25T08:45:16.318 INFO:tasks.rados.rados.0.trial169.stdout:772: oid 50 version 405 is already newer than 405 2026-02-25T08:45:16.318 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 50 v 405 (ObjNum 369 snap 100 seq_num 369) dirty exists 2026-02-25T08:45:16.318 INFO:tasks.rados.rados.0.trial169.stdout:772: left oid 50 (ObjNum 369 snap 100 seq_num 369) 2026-02-25T08:45:16.510 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:16.586 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:16.584+0000 7f4ae6ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:16.589 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:16.587+0000 7f4ae6ddc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:16.592 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:16.630 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:16.631 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:17.206 INFO:tasks.rados.rados.0.trial169.stdout:771: done (6 left) 2026-02-25T08:45:17.206 INFO:tasks.rados.rados.0.trial169.stdout:772: done (5 left) 2026-02-25T08:45:17.206 INFO:tasks.rados.rados.0.trial169.stdout:773: done (4 left) 2026-02-25T08:45:17.206 INFO:tasks.rados.rados.0.trial169.stdout:774: done (3 left) 2026-02-25T08:45:17.207 INFO:tasks.rados.rados.0.trial169.stdout:775: done (2 left) 2026-02-25T08:45:17.207 INFO:tasks.rados.rados.0.trial169.stdout:776: done (1 left) 2026-02-25T08:45:17.207 INFO:tasks.rados.rados.0.trial169.stdout:778: done (0 left) 2026-02-25T08:45:17.207 INFO:tasks.rados.rados.0.trial169.stdout:779: write oid 13 current snap is 101 2026-02-25T08:45:17.207 INFO:tasks.rados.rados.0.trial169.stdout:779: seq_num 372 ranges {652204=777572,1952125=300079} 2026-02-25T08:45:17.214 INFO:tasks.rados.rados.0.trial169.stdout:779: writing trial16941936-13 from 652204 to 1429776 tid 1 2026-02-25T08:45:17.219 INFO:tasks.rados.rados.0.trial169.stdout:779: writing trial16941936-13 from 1952125 to 2252204 tid 2 2026-02-25T08:45:17.219 INFO:tasks.rados.rados.0.trial169.stdout:780: write (excl) oid 39 current snap is 101 2026-02-25T08:45:17.219 INFO:tasks.rados.rados.0.trial169.stdout:780: seq_num 373 ranges {669011=569174,2010607=258404} 2026-02-25T08:45:17.225 INFO:tasks.rados.rados.0.trial169.stdout:780: writing trial16941936-39 from 669011 to 1238185 tid 1 2026-02-25T08:45:17.232 INFO:tasks.rados.rados.0.trial169.stdout:780: writing trial16941936-39 from 2010607 to 2269011 tid 2 2026-02-25T08:45:17.232 INFO:tasks.rados.rados.0.trial169.stdout:781: read oid 37 snap -1 2026-02-25T08:45:17.232 INFO:tasks.rados.rados.0.trial169.stdout:781: expect (ObjNum 356 snap 96 seq_num 356) 2026-02-25T08:45:17.232 INFO:tasks.rados.rados.0.trial169.stdout:782: delete oid 50 current snap is 101 2026-02-25T08:45:17.261 INFO:tasks.rados.rados.0.trial169.stdout:781: expect (ObjNum 356 snap 96 seq_num 356) 2026-02-25T08:45:17.298 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:17.344 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 492395 2026-02-25T08:45:17.388 INFO:tasks.rados.rados.0.trial169.stdout:779: finishing write tid 1 to trial16941936-13 2026-02-25T08:45:17.388 INFO:tasks.rados.rados.0.trial169.stdout:781: done (3 left) 2026-02-25T08:45:17.388 INFO:tasks.rados.rados.0.trial169.stdout:782: done (2 left) 2026-02-25T08:45:17.388 INFO:tasks.rados.rados.0.trial169.stdout:783: snap_create 2026-02-25T08:45:17.388 INFO:tasks.rados.rados.0.trial169.stdout:779: finishing write tid 2 to trial16941936-13 2026-02-25T08:45:17.388 INFO:tasks.rados.rados.0.trial169.stdout:779: finishing write tid 3 to trial16941936-13 2026-02-25T08:45:17.388 INFO:tasks.rados.rados.0.trial169.stdout:779: finishing write tid 4 to trial16941936-13 2026-02-25T08:45:17.388 INFO:tasks.rados.rados.0.trial169.stdout:779: oid 13 updating version 0 to 320 2026-02-25T08:45:17.388 INFO:tasks.rados.rados.0.trial169.stdout:779: oid 13 version 320 is already newer than 320 2026-02-25T08:45:17.388 INFO:tasks.rados.rados.0.trial169.stdout:779: oid 13 version 320 is already newer than 320 2026-02-25T08:45:17.389 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 13 v 320 (ObjNum 372 snap 101 seq_num 372) dirty exists 2026-02-25T08:45:17.389 INFO:tasks.rados.rados.0.trial169.stdout:779: left oid 13 (ObjNum 372 snap 101 seq_num 372) 2026-02-25T08:45:17.389 INFO:tasks.rados.rados.0.trial169.stdout:780: finishing write tid 1 to trial16941936-39 2026-02-25T08:45:17.389 INFO:tasks.rados.rados.0.trial169.stdout:780: finishing write tid 2 to trial16941936-39 2026-02-25T08:45:17.389 INFO:tasks.rados.rados.0.trial169.stdout:780: finishing write tid 3 to trial16941936-39 2026-02-25T08:45:17.389 INFO:tasks.rados.rados.0.trial169.stdout:780: finishing write tid 4 to trial16941936-39 2026-02-25T08:45:17.389 INFO:tasks.rados.rados.0.trial169.stdout:780: oid 39 updating version 0 to 127 2026-02-25T08:45:17.389 INFO:tasks.rados.rados.0.trial169.stdout:780: oid 39 version 127 is already newer than 125 2026-02-25T08:45:17.389 INFO:tasks.rados.rados.0.trial169.stdout:780: oid 39 version 127 is already newer than 127 2026-02-25T08:45:17.389 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 39 v 127 (ObjNum 373 snap 101 seq_num 373) dirty exists 2026-02-25T08:45:17.389 INFO:tasks.rados.rados.0.trial169.stdout:780: left oid 39 (ObjNum 373 snap 101 seq_num 373) 2026-02-25T08:45:18.148 INFO:tasks.rados.rados.0.trial169.stdout:779: done (2 left) 2026-02-25T08:45:18.149 INFO:tasks.rados.rados.0.trial169.stdout:780: done (1 left) 2026-02-25T08:45:18.149 INFO:tasks.rados.rados.0.trial169.stdout:783: done (0 left) 2026-02-25T08:45:18.149 INFO:tasks.rados.rados.0.trial169.stdout:784: delete oid 17 current snap is 102 2026-02-25T08:45:18.172 INFO:tasks.rados.rados.0.trial169.stdout:784: done (0 left) 2026-02-25T08:45:18.172 INFO:tasks.rados.rados.0.trial169.stdout:785: write oid 1 current snap is 102 2026-02-25T08:45:18.172 INFO:tasks.rados.rados.0.trial169.stdout:785: seq_num 376 ranges {719432=794921,2186053=133379} 2026-02-25T08:45:18.178 INFO:tasks.rados.rados.0.trial169.stdout:785: writing trial16941936-1 from 719432 to 1514353 tid 1 2026-02-25T08:45:18.181 INFO:tasks.rados.rados.0.trial169.stdout:785: writing trial16941936-1 from 2186053 to 2319432 tid 2 2026-02-25T08:45:18.181 INFO:tasks.rados.rados.0.trial169.stdout:786: snap_remove snap 99 2026-02-25T08:45:18.632 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:18.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~193s 2026-02-25T08:45:18.709 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:18.707+0000 7f697ee8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:18.712 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:18.710+0000 7f697ee8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:18.715 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:18.754 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:18.755 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:19.151 INFO:tasks.rados.rados.0.trial169.stdout:786: done (1 left) 2026-02-25T08:45:19.151 INFO:tasks.rados.rados.0.trial169.stdout:787: read oid 7 snap 98 2026-02-25T08:45:19.151 INFO:tasks.rados.rados.0.trial169.stdout:787: expect deleted 2026-02-25T08:45:19.151 INFO:tasks.rados.rados.0.trial169.stdout:788: read oid 4 snap 96 2026-02-25T08:45:19.151 INFO:tasks.rados.rados.0.trial169.stdout:788: expect (ObjNum 278 snap 76 seq_num 278) 2026-02-25T08:45:19.152 INFO:tasks.rados.rados.0.trial169.stdout:789: read oid 46 snap 96 2026-02-25T08:45:19.152 INFO:tasks.rados.rados.0.trial169.stdout:789: expect deleted 2026-02-25T08:45:19.152 INFO:tasks.rados.rados.0.trial169.stdout:790: delete oid 31 current snap is 102 2026-02-25T08:45:19.152 INFO:tasks.rados.rados.0.trial169.stdout:785: finishing write tid 1 to trial16941936-1 2026-02-25T08:45:19.152 INFO:tasks.rados.rados.0.trial169.stdout:785: finishing write tid 2 to trial16941936-1 2026-02-25T08:45:19.152 INFO:tasks.rados.rados.0.trial169.stdout:785: finishing write tid 3 to trial16941936-1 2026-02-25T08:45:19.152 INFO:tasks.rados.rados.0.trial169.stdout:785: finishing write tid 4 to trial16941936-1 2026-02-25T08:45:19.152 INFO:tasks.rados.rados.0.trial169.stdout:785: oid 1 updating version 0 to 82 2026-02-25T08:45:19.152 INFO:tasks.rados.rados.0.trial169.stdout:785: oid 1 version 82 is already newer than 82 2026-02-25T08:45:19.152 INFO:tasks.rados.rados.0.trial169.stdout:785: oid 1 version 82 is already newer than 82 2026-02-25T08:45:19.152 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 1 v 82 (ObjNum 376 snap 102 seq_num 376) dirty exists 2026-02-25T08:45:19.152 INFO:tasks.rados.rados.0.trial169.stdout:785: left oid 1 (ObjNum 376 snap 102 seq_num 376) 2026-02-25T08:45:19.189 INFO:tasks.rados.rados.0.trial169.stdout:788: expect (ObjNum 278 snap 76 seq_num 278) 2026-02-25T08:45:19.190 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:19.251 INFO:tasks.rados.rados.0.trial169.stdout:785: done (4 left) 2026-02-25T08:45:19.251 INFO:tasks.rados.rados.0.trial169.stdout:787: done (3 left) 2026-02-25T08:45:19.251 INFO:tasks.rados.rados.0.trial169.stdout:788: done (2 left) 2026-02-25T08:45:19.251 INFO:tasks.rados.rados.0.trial169.stdout:789: done (1 left) 2026-02-25T08:45:19.251 INFO:tasks.rados.rados.0.trial169.stdout:790: done (0 left) 2026-02-25T08:45:19.251 INFO:tasks.rados.rados.0.trial169.stdout:791: write (excl) oid 7 current snap is 102 2026-02-25T08:45:19.251 INFO:tasks.rados.rados.0.trial169.stdout:791: seq_num 378 ranges {753046=545419,1786664=431793} 2026-02-25T08:45:19.256 INFO:tasks.rados.rados.0.trial169.stdout:791: writing trial16941936-7 from 753046 to 1298465 tid 1 2026-02-25T08:45:19.260 INFO:tasks.rados.rados.0.trial169.stdout:791: writing trial16941936-7 from 1786664 to 2218457 tid 2 2026-02-25T08:45:19.260 INFO:tasks.rados.rados.0.trial169.stdout:792: write (excl) oid 19 current snap is 102 2026-02-25T08:45:19.260 INFO:tasks.rados.rados.0.trial169.stdout:792: seq_num 379 ranges {769853=620668,1845146=524707} 2026-02-25T08:45:19.265 INFO:tasks.rados.rados.0.trial169.stdout:792: writing trial16941936-19 from 769853 to 1390521 tid 1 2026-02-25T08:45:19.269 INFO:tasks.rados.rados.0.trial169.stdout:792: writing trial16941936-19 from 1845146 to 2369853 tid 2 2026-02-25T08:45:19.269 INFO:tasks.rados.rados.0.trial169.stdout:793: read oid 46 snap -1 2026-02-25T08:45:19.269 INFO:tasks.rados.rados.0.trial169.stdout:793: expect deleted 2026-02-25T08:45:19.269 INFO:tasks.rados.rados.0.trial169.stdout:794: delete oid 27 current snap is 102 2026-02-25T08:45:19.277 INFO:tasks.rados.rados.0.trial169.stdout:791: finishing write tid 1 to trial16941936-7 2026-02-25T08:45:19.299 INFO:tasks.rados.rados.0.trial169.stdout:791: finishing write tid 2 to trial16941936-7 2026-02-25T08:45:19.305 INFO:tasks.rados.rados.0.trial169.stdout:791: finishing write tid 3 to trial16941936-7 2026-02-25T08:45:19.307 INFO:tasks.rados.rados.0.trial169.stdout:791: finishing write tid 4 to trial16941936-7 2026-02-25T08:45:19.307 INFO:tasks.rados.rados.0.trial169.stdout:791: oid 7 updating version 0 to 416 2026-02-25T08:45:19.307 INFO:tasks.rados.rados.0.trial169.stdout:791: oid 7 updating version 416 to 418 2026-02-25T08:45:19.307 INFO:tasks.rados.rados.0.trial169.stdout:791: oid 7 version 418 is already newer than 418 2026-02-25T08:45:19.307 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 7 v 418 (ObjNum 378 snap 102 seq_num 378) dirty exists 2026-02-25T08:45:19.307 INFO:tasks.rados.rados.0.trial169.stdout:791: left oid 7 (ObjNum 378 snap 102 seq_num 378) 2026-02-25T08:45:19.307 INFO:tasks.rados.rados.0.trial169.stdout:791: done (3 left) 2026-02-25T08:45:19.307 INFO:tasks.rados.rados.0.trial169.stdout:793: done (2 left) 2026-02-25T08:45:19.307 INFO:tasks.rados.rados.0.trial169.stdout:794: done (1 left) 2026-02-25T08:45:19.307 INFO:tasks.rados.rados.0.trial169.stdout:795: read oid 21 snap 95 2026-02-25T08:45:19.308 INFO:tasks.rados.rados.0.trial169.stdout:795: expect deleted 2026-02-25T08:45:19.308 INFO:tasks.rados.rados.0.trial169.stdout:796: write (excl) oid 1 current snap is 102 2026-02-25T08:45:19.308 INFO:tasks.rados.rados.0.trial169.stdout:796: seq_num 381 ranges {403467=487519,1562110=579120} 2026-02-25T08:45:19.310 INFO:tasks.rados.rados.0.trial169.stdout:796: writing trial16941936-1 from 403467 to 890986 tid 1 2026-02-25T08:45:19.315 INFO:tasks.rados.rados.0.trial169.stdout:796: writing trial16941936-1 from 1562110 to 2141230 tid 2 2026-02-25T08:45:19.315 INFO:tasks.rados.rados.0.trial169.stdout:797: snap_create 2026-02-25T08:45:19.324 INFO:tasks.rados.rados.0.trial169.stdout:792: finishing write tid 1 to trial16941936-19 2026-02-25T08:45:19.334 INFO:tasks.rados.rados.0.trial169.stdout:796: finishing write tid 1 to trial16941936-1 2026-02-25T08:45:19.349 INFO:tasks.rados.rados.0.trial169.stdout:796: finishing write tid 2 to trial16941936-1 2026-02-25T08:45:19.356 INFO:tasks.rados.rados.0.trial169.stdout:796: finishing write tid 3 to trial16941936-1 2026-02-25T08:45:19.359 INFO:tasks.rados.rados.0.trial169.stdout:796: finishing write tid 4 to trial16941936-1 2026-02-25T08:45:19.359 INFO:tasks.rados.rados.0.trial169.stdout:796: oid 1 updating version 0 to 83 2026-02-25T08:45:19.359 INFO:tasks.rados.rados.0.trial169.stdout:796: oid 1 updating version 83 to 85 2026-02-25T08:45:19.360 INFO:tasks.rados.rados.0.trial169.stdout:796: oid 1 version 85 is already newer than 85 2026-02-25T08:45:19.360 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 1 v 85 (ObjNum 381 snap 102 seq_num 381) dirty exists 2026-02-25T08:45:19.360 INFO:tasks.rados.rados.0.trial169.stdout:796: left oid 1 (ObjNum 381 snap 102 seq_num 381) 2026-02-25T08:45:19.366 INFO:tasks.rados.rados.0.trial169.stdout:792: finishing write tid 2 to trial16941936-19 2026-02-25T08:45:19.368 INFO:tasks.rados.rados.0.trial169.stdout:792: finishing write tid 3 to trial16941936-19 2026-02-25T08:45:19.370 INFO:tasks.rados.rados.0.trial169.stdout:792: finishing write tid 4 to trial16941936-19 2026-02-25T08:45:19.370 INFO:tasks.rados.rados.0.trial169.stdout:792: oid 19 updating version 0 to 120 2026-02-25T08:45:19.370 INFO:tasks.rados.rados.0.trial169.stdout:792: oid 19 version 120 is already newer than 118 2026-02-25T08:45:19.370 INFO:tasks.rados.rados.0.trial169.stdout:792: oid 19 version 120 is already newer than 120 2026-02-25T08:45:19.370 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 19 v 120 (ObjNum 379 snap 102 seq_num 379) dirty exists 2026-02-25T08:45:19.370 INFO:tasks.rados.rados.0.trial169.stdout:792: left oid 19 (ObjNum 379 snap 102 seq_num 379) 2026-02-25T08:45:20.214 INFO:tasks.rados.rados.0.trial169.stdout:792: done (3 left) 2026-02-25T08:45:20.215 INFO:tasks.rados.rados.0.trial169.stdout:795: done (2 left) 2026-02-25T08:45:20.215 INFO:tasks.rados.rados.0.trial169.stdout:796: done (1 left) 2026-02-25T08:45:20.215 INFO:tasks.rados.rados.0.trial169.stdout:797: done (0 left) 2026-02-25T08:45:20.215 INFO:tasks.rados.rados.0.trial169.stdout:798: write oid 2 current snap is 103 2026-02-25T08:45:20.215 INFO:tasks.rados.rados.0.trial169.stdout:798: seq_num 382 ranges {420274=562768,1620592=439131} 2026-02-25T08:45:20.219 INFO:tasks.rados.rados.0.trial169.stdout:798: writing trial16941936-2 from 420274 to 983042 tid 1 2026-02-25T08:45:20.224 INFO:tasks.rados.rados.0.trial169.stdout:798: writing trial16941936-2 from 1620592 to 2059723 tid 2 2026-02-25T08:45:20.224 INFO:tasks.rados.rados.0.trial169.stdout:799: read oid 6 snap -1 2026-02-25T08:45:20.224 INFO:tasks.rados.rados.0.trial169.stdout:799: expect (ObjNum 288 snap 79 seq_num 288) 2026-02-25T08:45:20.224 INFO:tasks.rados.rados.0.trial169.stdout:800: snap_remove snap 100 2026-02-25T08:45:20.319 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:45:20.390 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:20.392 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:20.393 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:45:20.755 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:20.829 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:20.826+0000 7f46d470b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:20.831 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:20.830+0000 7f46d470b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:20.834 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:20.874 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:20.874 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:21.218 INFO:tasks.rados.rados.0.trial169.stdout:800: done (2 left) 2026-02-25T08:45:21.218 INFO:tasks.rados.rados.0.trial169.stdout:801: snap_create 2026-02-25T08:45:21.218 INFO:tasks.rados.rados.0.trial169.stdout:799: expect (ObjNum 288 snap 79 seq_num 288) 2026-02-25T08:45:21.265 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:21.289 INFO:tasks.rados.rados.0.trial169.stdout:798: finishing write tid 1 to trial16941936-2 2026-02-25T08:45:21.289 INFO:tasks.rados.rados.0.trial169.stdout:798: finishing write tid 2 to trial16941936-2 2026-02-25T08:45:21.289 INFO:tasks.rados.rados.0.trial169.stdout:798: finishing write tid 3 to trial16941936-2 2026-02-25T08:45:21.289 INFO:tasks.rados.rados.0.trial169.stdout:798: finishing write tid 4 to trial16941936-2 2026-02-25T08:45:21.289 INFO:tasks.rados.rados.0.trial169.stdout:798: oid 2 updating version 0 to 65 2026-02-25T08:45:21.289 INFO:tasks.rados.rados.0.trial169.stdout:798: oid 2 version 65 is already newer than 65 2026-02-25T08:45:21.289 INFO:tasks.rados.rados.0.trial169.stdout:798: oid 2 version 65 is already newer than 65 2026-02-25T08:45:21.290 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 2 v 65 (ObjNum 382 snap 103 seq_num 382) dirty exists 2026-02-25T08:45:21.290 INFO:tasks.rados.rados.0.trial169.stdout:798: left oid 2 (ObjNum 382 snap 103 seq_num 382) 2026-02-25T08:45:22.223 INFO:tasks.rados.rados.0.trial169.stdout:798: done (2 left) 2026-02-25T08:45:22.223 INFO:tasks.rados.rados.0.trial169.stdout:799: done (1 left) 2026-02-25T08:45:22.223 INFO:tasks.rados.rados.0.trial169.stdout:801: done (0 left) 2026-02-25T08:45:22.223 INFO:tasks.rados.rados.0.trial169.stdout:802: write (excl) oid 40 current snap is 104 2026-02-25T08:45:22.223 INFO:tasks.rados.rados.0.trial169.stdout:802: seq_num 383 ranges {437081=638017,1679074=582789} 2026-02-25T08:45:22.227 INFO:tasks.rados.rados.0.trial169.stdout:802: writing trial16941936-40 from 437081 to 1075098 tid 1 2026-02-25T08:45:22.232 INFO:tasks.rados.rados.0.trial169.stdout:802: writing trial16941936-40 from 1679074 to 2261863 tid 2 2026-02-25T08:45:22.232 INFO:tasks.rados.rados.0.trial169.stdout:803: write (excl) oid 48 current snap is 104 2026-02-25T08:45:22.232 INFO:tasks.rados.rados.0.trial169.stdout:803: seq_num 384 ranges {453888=713266,1737556=442800} 2026-02-25T08:45:22.236 INFO:tasks.rados.rados.0.trial169.stdout:803: writing trial16941936-48 from 453888 to 1167154 tid 1 2026-02-25T08:45:22.240 INFO:tasks.rados.rados.0.trial169.stdout:803: writing trial16941936-48 from 1737556 to 2180356 tid 2 2026-02-25T08:45:22.240 INFO:tasks.rados.rados.0.trial169.stdout:804: write (excl) oid 13 current snap is 104 2026-02-25T08:45:22.240 INFO:tasks.rados.rados.0.trial169.stdout:804: seq_num 385 ranges {470695=788515,1679685=586458} 2026-02-25T08:45:22.245 INFO:tasks.rados.rados.0.trial169.stdout:804: writing trial16941936-13 from 470695 to 1259210 tid 1 2026-02-25T08:45:22.249 INFO:tasks.rados.rados.0.trial169.stdout:804: writing trial16941936-13 from 1679685 to 2266143 tid 2 2026-02-25T08:45:22.249 INFO:tasks.rados.rados.0.trial169.stdout:805: write (excl) oid 43 current snap is 104 2026-02-25T08:45:22.250 INFO:tasks.rados.rados.0.trial169.stdout:805: seq_num 386 ranges {487502=463764,1738167=446469} 2026-02-25T08:45:22.253 INFO:tasks.rados.rados.0.trial169.stdout:805: writing trial16941936-43 from 487502 to 951266 tid 1 2026-02-25T08:45:22.258 INFO:tasks.rados.rados.0.trial169.stdout:805: writing trial16941936-43 from 1738167 to 2184636 tid 2 2026-02-25T08:45:22.258 INFO:tasks.rados.rados.0.trial169.stdout:806: snap_remove snap 92 2026-02-25T08:45:22.875 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:22.953 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:22.950+0000 7f7d26c46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:22.955 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:22.954+0000 7f7d26c46640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:22.959 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:22.998 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:23.225 INFO:tasks.rados.rados.0.trial169.stdout:806: done (4 left) 2026-02-25T08:45:23.225 INFO:tasks.rados.rados.0.trial169.stdout:807: write oid 3 current snap is 104 2026-02-25T08:45:23.225 INFO:tasks.rados.rados.0.trial169.stdout:802: finishing write tid 1 to trial16941936-40 2026-02-25T08:45:23.225 INFO:tasks.rados.rados.0.trial169.stdout:807: seq_num 387 ranges {504309=539013,1796649=590127} 2026-02-25T08:45:23.229 INFO:tasks.rados.rados.0.trial169.stdout:807: writing trial16941936-3 from 504309 to 1043322 tid 1 2026-02-25T08:45:23.236 INFO:tasks.rados.rados.0.trial169.stdout:807: writing trial16941936-3 from 1796649 to 2386776 tid 2 2026-02-25T08:45:23.236 INFO:tasks.rados.rados.0.trial169.stdout:808: snap_create 2026-02-25T08:45:23.236 INFO:tasks.rados.rados.0.trial169.stdout:802: finishing write tid 2 to trial16941936-40 2026-02-25T08:45:23.236 INFO:tasks.rados.rados.0.trial169.stdout:804: finishing write tid 1 to trial16941936-13 2026-02-25T08:45:23.236 INFO:tasks.rados.rados.0.trial169.stdout:802: finishing write tid 3 to trial16941936-40 2026-02-25T08:45:23.236 INFO:tasks.rados.rados.0.trial169.stdout:802: finishing write tid 4 to trial16941936-40 2026-02-25T08:45:23.236 INFO:tasks.rados.rados.0.trial169.stdout:802: oid 40 updating version 0 to 423 2026-02-25T08:45:23.237 INFO:tasks.rados.rados.0.trial169.stdout:802: oid 40 version 423 is already newer than 423 2026-02-25T08:45:23.237 INFO:tasks.rados.rados.0.trial169.stdout:802: oid 40 version 423 is already newer than 421 2026-02-25T08:45:23.237 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 40 v 423 (ObjNum 383 snap 104 seq_num 383) dirty exists 2026-02-25T08:45:23.237 INFO:tasks.rados.rados.0.trial169.stdout:802: left oid 40 (ObjNum 383 snap 104 seq_num 383) 2026-02-25T08:45:23.237 INFO:tasks.rados.rados.0.trial169.stdout:803: finishing write tid 1 to trial16941936-48 2026-02-25T08:45:23.237 INFO:tasks.rados.rados.0.trial169.stdout:804: finishing write tid 2 to trial16941936-13 2026-02-25T08:45:23.237 INFO:tasks.rados.rados.0.trial169.stdout:804: finishing write tid 3 to trial16941936-13 2026-02-25T08:45:23.237 INFO:tasks.rados.rados.0.trial169.stdout:803: finishing write tid 2 to trial16941936-48 2026-02-25T08:45:23.237 INFO:tasks.rados.rados.0.trial169.stdout:803: finishing write tid 3 to trial16941936-48 2026-02-25T08:45:23.237 INFO:tasks.rados.rados.0.trial169.stdout:803: finishing write tid 4 to trial16941936-48 2026-02-25T08:45:23.237 INFO:tasks.rados.rados.0.trial169.stdout:803: oid 48 updating version 0 to 24 2026-02-25T08:45:23.237 INFO:tasks.rados.rados.0.trial169.stdout:803: oid 48 version 24 is already newer than 22 2026-02-25T08:45:23.237 INFO:tasks.rados.rados.0.trial169.stdout:803: oid 48 version 24 is already newer than 24 2026-02-25T08:45:23.238 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 48 v 24 (ObjNum 384 snap 104 seq_num 384) dirty exists 2026-02-25T08:45:23.238 INFO:tasks.rados.rados.0.trial169.stdout:803: left oid 48 (ObjNum 384 snap 104 seq_num 384) 2026-02-25T08:45:23.238 INFO:tasks.rados.rados.0.trial169.stdout:804: finishing write tid 4 to trial16941936-13 2026-02-25T08:45:23.238 INFO:tasks.rados.rados.0.trial169.stdout:804: oid 13 updating version 0 to 328 2026-02-25T08:45:23.238 INFO:tasks.rados.rados.0.trial169.stdout:804: oid 13 updating version 328 to 332 2026-02-25T08:45:23.238 INFO:tasks.rados.rados.0.trial169.stdout:804: oid 13 version 332 is already newer than 332 2026-02-25T08:45:23.238 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 13 v 332 (ObjNum 385 snap 104 seq_num 385) dirty exists 2026-02-25T08:45:23.238 INFO:tasks.rados.rados.0.trial169.stdout:804: left oid 13 (ObjNum 385 snap 104 seq_num 385) 2026-02-25T08:45:23.238 INFO:tasks.rados.rados.0.trial169.stdout:805: finishing write tid 1 to trial16941936-43 2026-02-25T08:45:23.238 INFO:tasks.rados.rados.0.trial169.stdout:805: finishing write tid 2 to trial16941936-43 2026-02-25T08:45:23.238 INFO:tasks.rados.rados.0.trial169.stdout:805: finishing write tid 3 to trial16941936-43 2026-02-25T08:45:23.238 INFO:tasks.rados.rados.0.trial169.stdout:805: finishing write tid 4 to trial16941936-43 2026-02-25T08:45:23.239 INFO:tasks.rados.rados.0.trial169.stdout:805: oid 43 updating version 0 to 81 2026-02-25T08:45:23.239 INFO:tasks.rados.rados.0.trial169.stdout:805: oid 43 version 81 is already newer than 81 2026-02-25T08:45:23.239 INFO:tasks.rados.rados.0.trial169.stdout:805: oid 43 version 81 is already newer than 79 2026-02-25T08:45:23.239 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 43 v 81 (ObjNum 386 snap 104 seq_num 386) dirty exists 2026-02-25T08:45:23.239 INFO:tasks.rados.rados.0.trial169.stdout:805: left oid 43 (ObjNum 386 snap 104 seq_num 386) 2026-02-25T08:45:23.287 INFO:tasks.rados.rados.0.trial169.stdout:807: finishing write tid 1 to trial16941936-3 2026-02-25T08:45:23.293 INFO:tasks.rados.rados.0.trial169.stdout:807: finishing write tid 2 to trial16941936-3 2026-02-25T08:45:23.303 INFO:tasks.rados.rados.0.trial169.stdout:807: finishing write tid 3 to trial16941936-3 2026-02-25T08:45:23.305 INFO:tasks.rados.rados.0.trial169.stdout:807: finishing write tid 4 to trial16941936-3 2026-02-25T08:45:23.305 INFO:tasks.rados.rados.0.trial169.stdout:807: oid 3 updating version 0 to 165 2026-02-25T08:45:23.305 INFO:tasks.rados.rados.0.trial169.stdout:807: oid 3 version 165 is already newer than 165 2026-02-25T08:45:23.305 INFO:tasks.rados.rados.0.trial169.stdout:807: oid 3 version 165 is already newer than 165 2026-02-25T08:45:23.305 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 3 v 165 (ObjNum 387 snap 104 seq_num 387) dirty exists 2026-02-25T08:45:23.305 INFO:tasks.rados.rados.0.trial169.stdout:807: left oid 3 (ObjNum 387 snap 104 seq_num 387) 2026-02-25T08:45:24.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~198s 2026-02-25T08:45:24.230 INFO:tasks.rados.rados.0.trial169.stdout:802: done (5 left) 2026-02-25T08:45:24.230 INFO:tasks.rados.rados.0.trial169.stdout:803: done (4 left) 2026-02-25T08:45:24.230 INFO:tasks.rados.rados.0.trial169.stdout:804: done (3 left) 2026-02-25T08:45:24.230 INFO:tasks.rados.rados.0.trial169.stdout:805: done (2 left) 2026-02-25T08:45:24.230 INFO:tasks.rados.rados.0.trial169.stdout:807: done (1 left) 2026-02-25T08:45:24.230 INFO:tasks.rados.rados.0.trial169.stdout:808: done (0 left) 2026-02-25T08:45:24.230 INFO:tasks.rados.rados.0.trial169.stdout:809: snap_create 2026-02-25T08:45:24.999 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:25.076 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:25.073+0000 7ff7cc9a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:25.079 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:25.077+0000 7ff7cc9a4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:25.082 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:25.124 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:25.125 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:25.234 INFO:tasks.rados.rados.0.trial169.stdout:809: done (0 left) 2026-02-25T08:45:25.234 INFO:tasks.rados.rados.0.trial169.stdout:810: read oid 17 snap -1 2026-02-25T08:45:25.234 INFO:tasks.rados.rados.0.trial169.stdout:810: expect deleted 2026-02-25T08:45:25.234 INFO:tasks.rados.rados.0.trial169.stdout:811: read oid 28 snap 93 2026-02-25T08:45:25.235 INFO:tasks.rados.rados.0.trial169.stdout:811: expect (ObjNum 319 snap 86 seq_num 319) 2026-02-25T08:45:25.235 INFO:tasks.rados.rados.0.trial169.stdout:812: write oid 46 current snap is 106 2026-02-25T08:45:25.235 INFO:tasks.rados.rados.0.trial169.stdout:812: seq_num 388 ranges {521116=730615,1971484=549632} 2026-02-25T08:45:25.239 INFO:tasks.rados.rados.0.trial169.stdout:812: writing trial16941936-46 from 521116 to 1251731 tid 1 2026-02-25T08:45:25.244 INFO:tasks.rados.rados.0.trial169.stdout:812: writing trial16941936-46 from 1971484 to 2521116 tid 2 2026-02-25T08:45:25.244 INFO:tasks.rados.rados.0.trial169.stdout:813: delete oid 4 current snap is 106 2026-02-25T08:45:25.274 INFO:tasks.rados.rados.0.trial169.stdout:811: expect (ObjNum 319 snap 86 seq_num 319) 2026-02-25T08:45:25.319 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 79385 2026-02-25T08:45:25.394 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:45:25.399 INFO:tasks.rados.rados.0.trial169.stdout:812: finishing write tid 1 to trial16941936-46 2026-02-25T08:45:25.399 INFO:tasks.rados.rados.0.trial169.stdout:812: finishing write tid 2 to trial16941936-46 2026-02-25T08:45:25.399 INFO:tasks.rados.rados.0.trial169.stdout:812: finishing write tid 3 to trial16941936-46 2026-02-25T08:45:25.399 INFO:tasks.rados.rados.0.trial169.stdout:810: done (3 left) 2026-02-25T08:45:25.399 INFO:tasks.rados.rados.0.trial169.stdout:811: done (2 left) 2026-02-25T08:45:25.399 INFO:tasks.rados.rados.0.trial169.stdout:813: done (1 left) 2026-02-25T08:45:25.400 INFO:tasks.rados.rados.0.trial169.stdout:814: snap_remove snap 104 2026-02-25T08:45:25.464 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:25.467 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:25.468 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:45:26.238 INFO:tasks.rados.rados.0.trial169.stdout:814: done (1 left) 2026-02-25T08:45:26.238 INFO:tasks.rados.rados.0.trial169.stdout:815: snap_create 2026-02-25T08:45:26.238 INFO:tasks.rados.rados.0.trial169.stdout:812: finishing write tid 4 to trial16941936-46 2026-02-25T08:45:26.238 INFO:tasks.rados.rados.0.trial169.stdout:812: oid 46 updating version 0 to 136 2026-02-25T08:45:26.238 INFO:tasks.rados.rados.0.trial169.stdout:812: oid 46 version 136 is already newer than 136 2026-02-25T08:45:26.238 INFO:tasks.rados.rados.0.trial169.stdout:812: oid 46 version 136 is already newer than 136 2026-02-25T08:45:26.238 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 46 v 136 (ObjNum 388 snap 106 seq_num 388) dirty exists 2026-02-25T08:45:26.239 INFO:tasks.rados.rados.0.trial169.stdout:812: left oid 46 (ObjNum 388 snap 106 seq_num 388) 2026-02-25T08:45:27.126 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:27.204 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:27.201+0000 7f71bd10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:27.206 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:27.205+0000 7f71bd10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:27.210 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:27.238 INFO:tasks.rados.rados.0.trial169.stdout:812: done (1 left) 2026-02-25T08:45:27.238 INFO:tasks.rados.rados.0.trial169.stdout:815: done (0 left) 2026-02-25T08:45:27.238 INFO:tasks.rados.rados.0.trial169.stdout:816: write oid 48 current snap is 107 2026-02-25T08:45:27.238 INFO:tasks.rados.rados.0.trial169.stdout:816: seq_num 390 ranges {554730=481113,1572095=737454} 2026-02-25T08:45:27.242 INFO:tasks.rados.rados.0.trial169.stdout:816: writing trial16941936-48 from 554730 to 1035843 tid 1 2026-02-25T08:45:27.248 INFO:tasks.rados.rados.0.trial169.stdout:816: writing trial16941936-48 from 1572095 to 2309549 tid 2 2026-02-25T08:45:27.248 INFO:tasks.rados.rados.0.trial169.stdout:817: snap_remove snap 96 2026-02-25T08:45:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:27.261 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:28.151 INFO:tasks.rados.rados.0.trial169.stdout:817: done (1 left) 2026-02-25T08:45:28.152 INFO:tasks.rados.rados.0.trial169.stdout:818: read oid 12 snap 95 2026-02-25T08:45:28.152 INFO:tasks.rados.rados.0.trial169.stdout:818: expect (ObjNum 346 snap 95 seq_num 346) 2026-02-25T08:45:28.152 INFO:tasks.rados.rados.0.trial169.stdout:819: snap_remove snap 106 2026-02-25T08:45:29.152 INFO:tasks.rados.rados.0.trial169.stdout:819: done (2 left) 2026-02-25T08:45:29.153 INFO:tasks.rados.rados.0.trial169.stdout:820: write (excl) oid 49 current snap is 107 2026-02-25T08:45:29.153 INFO:tasks.rados.rados.0.trial169.stdout:816: finishing write tid 1 to trial16941936-48 2026-02-25T08:45:29.153 INFO:tasks.rados.rados.0.trial169.stdout:820: seq_num 391 ranges {571537=556362,1630577=597465} 2026-02-25T08:45:29.156 INFO:tasks.rados.rados.0.trial169.stdout:820: writing trial16941936-49 from 571537 to 1127899 tid 1 2026-02-25T08:45:29.162 INFO:tasks.rados.rados.0.trial169.stdout:820: writing trial16941936-49 from 1630577 to 2228042 tid 2 2026-02-25T08:45:29.162 INFO:tasks.rados.rados.0.trial169.stdout:821: read oid 28 snap -1 2026-02-25T08:45:29.162 INFO:tasks.rados.rados.0.trial169.stdout:821: expect (ObjNum 370 snap 100 seq_num 370) 2026-02-25T08:45:29.162 INFO:tasks.rados.rados.0.trial169.stdout:822: write oid 17 current snap is 107 2026-02-25T08:45:29.162 INFO:tasks.rados.rados.0.trial169.stdout:816: finishing write tid 2 to trial16941936-48 2026-02-25T08:45:29.162 INFO:tasks.rados.rados.0.trial169.stdout:816: finishing write tid 3 to trial16941936-48 2026-02-25T08:45:29.162 INFO:tasks.rados.rados.0.trial169.stdout:822: seq_num 392 ranges {588344=631611,1689059=741123} 2026-02-25T08:45:29.166 INFO:tasks.rados.rados.0.trial169.stdout:822: writing trial16941936-17 from 588344 to 1219955 tid 1 2026-02-25T08:45:29.172 INFO:tasks.rados.rados.0.trial169.stdout:822: writing trial16941936-17 from 1689059 to 2430182 tid 2 2026-02-25T08:45:29.172 INFO:tasks.rados.rados.0.trial169.stdout:823: snap_create 2026-02-25T08:45:29.172 INFO:tasks.rados.rados.0.trial169.stdout:816: finishing write tid 4 to trial16941936-48 2026-02-25T08:45:29.172 INFO:tasks.rados.rados.0.trial169.stdout:816: oid 48 updating version 0 to 28 2026-02-25T08:45:29.172 INFO:tasks.rados.rados.0.trial169.stdout:816: oid 48 version 28 is already newer than 28 2026-02-25T08:45:29.172 INFO:tasks.rados.rados.0.trial169.stdout:816: oid 48 version 28 is already newer than 28 2026-02-25T08:45:29.172 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 48 v 28 (ObjNum 390 snap 107 seq_num 390) dirty exists 2026-02-25T08:45:29.172 INFO:tasks.rados.rados.0.trial169.stdout:816: left oid 48 (ObjNum 390 snap 107 seq_num 390) 2026-02-25T08:45:29.173 INFO:tasks.rados.rados.0.trial169.stdout:818: expect (ObjNum 346 snap 95 seq_num 346) 2026-02-25T08:45:29.261 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:822: finishing write tid 1 to trial16941936-17 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:820: finishing write tid 1 to trial16941936-49 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:822: finishing write tid 2 to trial16941936-17 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:820: finishing write tid 2 to trial16941936-49 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:822: finishing write tid 3 to trial16941936-17 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:820: finishing write tid 3 to trial16941936-49 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:820: finishing write tid 4 to trial16941936-49 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:820: oid 49 updating version 0 to 350 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:820: oid 49 version 350 is already newer than 350 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:820: oid 49 version 350 is already newer than 348 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 49 v 350 (ObjNum 391 snap 107 seq_num 391) dirty exists 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:820: left oid 49 (ObjNum 391 snap 107 seq_num 391) 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:822: finishing write tid 4 to trial16941936-17 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:822: oid 17 updating version 0 to 68 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:822: oid 17 version 68 is already newer than 68 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:822: oid 17 version 68 is already newer than 68 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 17 v 68 (ObjNum 392 snap 107 seq_num 392) dirty exists 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:822: left oid 17 (ObjNum 392 snap 107 seq_num 392) 2026-02-25T08:45:29.319 INFO:tasks.rados.rados.0.trial169.stdout:821: expect (ObjNum 370 snap 100 seq_num 370) 2026-02-25T08:45:29.336 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:29.334+0000 7f1a53b9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:29.339 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:29.337+0000 7f1a53b9c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:29.342 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:29.382 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:29.390 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1623089 2026-02-25T08:45:29.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~204s 2026-02-25T08:45:30.155 INFO:tasks.rados.rados.0.trial169.stdout:816: done (5 left) 2026-02-25T08:45:30.155 INFO:tasks.rados.rados.0.trial169.stdout:818: done (4 left) 2026-02-25T08:45:30.155 INFO:tasks.rados.rados.0.trial169.stdout:820: done (3 left) 2026-02-25T08:45:30.155 INFO:tasks.rados.rados.0.trial169.stdout:821: done (2 left) 2026-02-25T08:45:30.155 INFO:tasks.rados.rados.0.trial169.stdout:822: done (1 left) 2026-02-25T08:45:30.156 INFO:tasks.rados.rados.0.trial169.stdout:823: done (0 left) 2026-02-25T08:45:30.156 INFO:tasks.rados.rados.0.trial169.stdout:824: write oid 34 current snap is 108 2026-02-25T08:45:30.156 INFO:tasks.rados.rados.0.trial169.stdout:824: seq_num 393 ranges {605151=706860,2031188=573963} 2026-02-25T08:45:30.160 INFO:tasks.rados.rados.0.trial169.stdout:824: writing trial16941936-34 from 605151 to 1312011 tid 1 2026-02-25T08:45:30.165 INFO:tasks.rados.rados.0.trial169.stdout:824: writing trial16941936-34 from 2031188 to 2605151 tid 2 2026-02-25T08:45:30.166 INFO:tasks.rados.rados.0.trial169.stdout:825: snap_create 2026-02-25T08:45:30.200 INFO:tasks.rados.rados.0.trial169.stdout:824: finishing write tid 1 to trial16941936-34 2026-02-25T08:45:30.206 INFO:tasks.rados.rados.0.trial169.stdout:824: finishing write tid 2 to trial16941936-34 2026-02-25T08:45:30.209 INFO:tasks.rados.rados.0.trial169.stdout:824: finishing write tid 3 to trial16941936-34 2026-02-25T08:45:30.211 INFO:tasks.rados.rados.0.trial169.stdout:824: finishing write tid 4 to trial16941936-34 2026-02-25T08:45:30.211 INFO:tasks.rados.rados.0.trial169.stdout:824: oid 34 updating version 0 to 124 2026-02-25T08:45:30.211 INFO:tasks.rados.rados.0.trial169.stdout:824: oid 34 version 124 is already newer than 124 2026-02-25T08:45:30.211 INFO:tasks.rados.rados.0.trial169.stdout:824: oid 34 version 124 is already newer than 124 2026-02-25T08:45:30.211 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 34 v 124 (ObjNum 393 snap 108 seq_num 393) dirty exists 2026-02-25T08:45:30.211 INFO:tasks.rados.rados.0.trial169.stdout:824: left oid 34 (ObjNum 393 snap 108 seq_num 393) 2026-02-25T08:45:30.470 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:45:30.539 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:30.542 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:45:31.157 INFO:tasks.rados.rados.0.trial169.stdout:824: done (1 left) 2026-02-25T08:45:31.157 INFO:tasks.rados.rados.0.trial169.stdout:825: done (0 left) 2026-02-25T08:45:31.157 INFO:tasks.rados.rados.0.trial169.stdout:826: snap_remove snap 107 2026-02-25T08:45:31.383 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:31.462 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:31.460+0000 7f3b9930b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:31.465 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:31.463+0000 7f3b9930b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:31.468 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:31.506 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:31.506 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:32.162 INFO:tasks.rados.rados.0.trial169.stdout:826: done (0 left) 2026-02-25T08:45:32.162 INFO:tasks.rados.rados.0.trial169.stdout:827: read oid 44 snap -1 2026-02-25T08:45:32.162 INFO:tasks.rados.rados.0.trial169.stdout:827: expect deleted 2026-02-25T08:45:32.162 INFO:tasks.rados.rados.0.trial169.stdout:828: snap_remove snap 101 2026-02-25T08:45:33.166 INFO:tasks.rados.rados.0.trial169.stdout:828: done (1 left) 2026-02-25T08:45:33.166 INFO:tasks.rados.rados.0.trial169.stdout:829: write oid 5 current snap is 109 2026-02-25T08:45:33.166 INFO:tasks.rados.rados.0.trial169.stdout:829: seq_num 394 ranges {621958=782109,2089670=532288} 2026-02-25T08:45:33.170 INFO:tasks.rados.rados.0.trial169.stdout:829: writing trial16941936-5 from 621958 to 1404067 tid 1 2026-02-25T08:45:33.176 INFO:tasks.rados.rados.0.trial169.stdout:829: writing trial16941936-5 from 2089670 to 2621958 tid 2 2026-02-25T08:45:33.176 INFO:tasks.rados.rados.0.trial169.stdout:830: read oid 26 snap -1 2026-02-25T08:45:33.176 INFO:tasks.rados.rados.0.trial169.stdout:830: expect (ObjNum 361 snap 98 seq_num 361) 2026-02-25T08:45:33.176 INFO:tasks.rados.rados.0.trial169.stdout:831: write (excl) oid 4 current snap is 109 2026-02-25T08:45:33.176 INFO:tasks.rados.rados.0.trial169.stdout:831: seq_num 395 ranges {638765=457358,1748152=604803} 2026-02-25T08:45:33.179 INFO:tasks.rados.rados.0.trial169.stdout:831: writing trial16941936-4 from 638765 to 1096123 tid 1 2026-02-25T08:45:33.184 INFO:tasks.rados.rados.0.trial169.stdout:831: writing trial16941936-4 from 1748152 to 2352955 tid 2 2026-02-25T08:45:33.184 INFO:tasks.rados.rados.0.trial169.stdout:832: write (excl) oid 33 current snap is 109 2026-02-25T08:45:33.185 INFO:tasks.rados.rados.0.trial169.stdout:832: seq_num 396 ranges {655572=648960,1922987=732585} 2026-02-25T08:45:33.189 INFO:tasks.rados.rados.0.trial169.stdout:832: writing trial16941936-33 from 655572 to 1304532 tid 1 2026-02-25T08:45:33.195 INFO:tasks.rados.rados.0.trial169.stdout:832: writing trial16941936-33 from 1922987 to 2655572 tid 2 2026-02-25T08:45:33.195 INFO:tasks.rados.rados.0.trial169.stdout:827: done (4 left) 2026-02-25T08:45:33.195 INFO:tasks.rados.rados.0.trial169.stdout:833: read oid 6 snap -1 2026-02-25T08:45:33.195 INFO:tasks.rados.rados.0.trial169.stdout:833: expect (ObjNum 288 snap 79 seq_num 288) 2026-02-25T08:45:33.195 INFO:tasks.rados.rados.0.trial169.stdout:834: delete oid 7 current snap is 109 2026-02-25T08:45:33.224 INFO:tasks.rados.rados.0.trial169.stdout:833: expect (ObjNum 288 snap 79 seq_num 288) 2026-02-25T08:45:33.294 INFO:tasks.rados.rados.0.trial169.stdout:831: finishing write tid 1 to trial16941936-4 2026-02-25T08:45:33.295 INFO:tasks.rados.rados.0.trial169.stdout:830: expect (ObjNum 361 snap 98 seq_num 361) 2026-02-25T08:45:33.348 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 517326 2026-02-25T08:45:33.394 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 948183 2026-02-25T08:45:33.432 INFO:tasks.rados.rados.0.trial169.stdout:831: finishing write tid 2 to trial16941936-4 2026-02-25T08:45:33.432 INFO:tasks.rados.rados.0.trial169.stdout:830: done (5 left) 2026-02-25T08:45:33.433 INFO:tasks.rados.rados.0.trial169.stdout:833: done (4 left) 2026-02-25T08:45:33.433 INFO:tasks.rados.rados.0.trial169.stdout:834: done (3 left) 2026-02-25T08:45:33.433 INFO:tasks.rados.rados.0.trial169.stdout:835: write (excl) oid 50 current snap is 109 2026-02-25T08:45:33.433 INFO:tasks.rados.rados.0.trial169.stdout:831: finishing write tid 3 to trial16941936-4 2026-02-25T08:45:33.433 INFO:tasks.rados.rados.0.trial169.stdout:831: finishing write tid 4 to trial16941936-4 2026-02-25T08:45:33.433 INFO:tasks.rados.rados.0.trial169.stdout:831: oid 4 updating version 0 to 446 2026-02-25T08:45:33.433 INFO:tasks.rados.rados.0.trial169.stdout:831: oid 4 version 446 is already newer than 446 2026-02-25T08:45:33.433 INFO:tasks.rados.rados.0.trial169.stdout:831: oid 4 version 446 is already newer than 444 2026-02-25T08:45:33.433 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 4 v 446 (ObjNum 395 snap 109 seq_num 395) dirty exists 2026-02-25T08:45:33.433 INFO:tasks.rados.rados.0.trial169.stdout:831: left oid 4 (ObjNum 395 snap 109 seq_num 395) 2026-02-25T08:45:33.433 INFO:tasks.rados.rados.0.trial169.stdout:829: finishing write tid 1 to trial16941936-5 2026-02-25T08:45:33.434 INFO:tasks.rados.rados.0.trial169.stdout:829: finishing write tid 2 to trial16941936-5 2026-02-25T08:45:33.434 INFO:tasks.rados.rados.0.trial169.stdout:835: seq_num 398 ranges {689186=799458,1923598=752130} 2026-02-25T08:45:33.437 INFO:tasks.rados.rados.0.trial169.stdout:835: writing trial16941936-50 from 689186 to 1488644 tid 1 2026-02-25T08:45:33.442 INFO:tasks.rados.rados.0.trial169.stdout:835: writing trial16941936-50 from 1923598 to 2675728 tid 2 2026-02-25T08:45:33.443 INFO:tasks.rados.rados.0.trial169.stdout:831: done (3 left) 2026-02-25T08:45:33.443 INFO:tasks.rados.rados.0.trial169.stdout:836: write oid 49 current snap is 109 2026-02-25T08:45:33.443 INFO:tasks.rados.rados.0.trial169.stdout:836: seq_num 399 ranges {705993=474707,1582080=612141} 2026-02-25T08:45:33.447 INFO:tasks.rados.rados.0.trial169.stdout:836: writing trial16941936-49 from 705993 to 1180700 tid 1 2026-02-25T08:45:33.451 INFO:tasks.rados.rados.0.trial169.stdout:836: writing trial16941936-49 from 1582080 to 2194221 tid 2 2026-02-25T08:45:33.451 INFO:tasks.rados.rados.0.trial169.stdout:837: write (excl) oid 19 current snap is 109 2026-02-25T08:45:33.451 INFO:tasks.rados.rados.0.trial169.stdout:837: seq_num 400 ranges {722800=549956,2040562=682238} 2026-02-25T08:45:33.456 INFO:tasks.rados.rados.0.trial169.stdout:837: writing trial16941936-19 from 722800 to 1272756 tid 1 2026-02-25T08:45:33.461 INFO:tasks.rados.rados.0.trial169.stdout:837: writing trial16941936-19 from 2040562 to 2722800 tid 2 2026-02-25T08:45:33.462 INFO:tasks.rados.rados.0.trial169.stdout:838: write (excl) oid 45 current snap is 109 2026-02-25T08:45:33.462 INFO:tasks.rados.rados.0.trial169.stdout:829: finishing write tid 3 to trial16941936-5 2026-02-25T08:45:33.462 INFO:tasks.rados.rados.0.trial169.stdout:838: seq_num 401 ranges {739607=625205,1982691=499457} 2026-02-25T08:45:33.466 INFO:tasks.rados.rados.0.trial169.stdout:838: writing trial16941936-45 from 739607 to 1364812 tid 1 2026-02-25T08:45:33.471 INFO:tasks.rados.rados.0.trial169.stdout:838: writing trial16941936-45 from 1982691 to 2482148 tid 2 2026-02-25T08:45:33.471 INFO:tasks.rados.rados.0.trial169.stdout:839: write oid 2 current snap is 109 2026-02-25T08:45:33.471 INFO:tasks.rados.rados.0.trial169.stdout:839: seq_num 402 ranges {756414=700454,2041173=715241} 2026-02-25T08:45:33.476 INFO:tasks.rados.rados.0.trial169.stdout:839: writing trial16941936-2 from 756414 to 1456868 tid 1 2026-02-25T08:45:33.482 INFO:tasks.rados.rados.0.trial169.stdout:839: writing trial16941936-2 from 2041173 to 2756414 tid 2 2026-02-25T08:45:33.482 INFO:tasks.rados.rados.0.trial169.stdout:840: write oid 36 current snap is 109 2026-02-25T08:45:33.483 INFO:tasks.rados.rados.0.trial169.stdout:829: finishing write tid 4 to trial16941936-5 2026-02-25T08:45:33.483 INFO:tasks.rados.rados.0.trial169.stdout:829: oid 5 updating version 0 to 175 2026-02-25T08:45:33.483 INFO:tasks.rados.rados.0.trial169.stdout:829: oid 5 version 175 is already newer than 175 2026-02-25T08:45:33.483 INFO:tasks.rados.rados.0.trial169.stdout:829: oid 5 version 175 is already newer than 175 2026-02-25T08:45:33.483 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 5 v 175 (ObjNum 394 snap 109 seq_num 394) dirty exists 2026-02-25T08:45:33.483 INFO:tasks.rados.rados.0.trial169.stdout:829: left oid 5 (ObjNum 394 snap 109 seq_num 394) 2026-02-25T08:45:33.483 INFO:tasks.rados.rados.0.trial169.stdout:832: finishing write tid 1 to trial16941936-33 2026-02-25T08:45:33.483 INFO:tasks.rados.rados.0.trial169.stdout:832: finishing write tid 2 to trial16941936-33 2026-02-25T08:45:33.483 INFO:tasks.rados.rados.0.trial169.stdout:832: finishing write tid 3 to trial16941936-33 2026-02-25T08:45:33.483 INFO:tasks.rados.rados.0.trial169.stdout:832: finishing write tid 4 to trial16941936-33 2026-02-25T08:45:33.483 INFO:tasks.rados.rados.0.trial169.stdout:832: oid 33 updating version 0 to 450 2026-02-25T08:45:33.483 INFO:tasks.rados.rados.0.trial169.stdout:832: oid 33 version 450 is already newer than 448 2026-02-25T08:45:33.484 INFO:tasks.rados.rados.0.trial169.stdout:832: oid 33 version 450 is already newer than 450 2026-02-25T08:45:33.484 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 33 v 450 (ObjNum 396 snap 109 seq_num 396) dirty exists 2026-02-25T08:45:33.484 INFO:tasks.rados.rados.0.trial169.stdout:832: left oid 33 (ObjNum 396 snap 109 seq_num 396) 2026-02-25T08:45:33.484 INFO:tasks.rados.rados.0.trial169.stdout:840: seq_num 403 ranges {773221=492056,1816008=503126} 2026-02-25T08:45:33.487 INFO:tasks.rados.rados.0.trial169.stdout:840: writing trial16941936-36 from 773221 to 1265277 tid 1 2026-02-25T08:45:33.491 INFO:tasks.rados.rados.0.trial169.stdout:840: writing trial16941936-36 from 1816008 to 2319134 tid 2 2026-02-25T08:45:33.492 INFO:tasks.rados.rados.0.trial169.stdout:829: done (7 left) 2026-02-25T08:45:33.492 INFO:tasks.rados.rados.0.trial169.stdout:832: done (6 left) 2026-02-25T08:45:33.492 INFO:tasks.rados.rados.0.trial169.stdout:841: read oid 23 snap -1 2026-02-25T08:45:33.492 INFO:tasks.rados.rados.0.trial169.stdout:841: expect (ObjNum 357 snap 97 seq_num 357) 2026-02-25T08:45:33.492 INFO:tasks.rados.rados.0.trial169.stdout:842: delete oid 13 current snap is 109 2026-02-25T08:45:33.492 INFO:tasks.rados.rados.0.trial169.stdout:835: finishing write tid 1 to trial16941936-50 2026-02-25T08:45:33.492 INFO:tasks.rados.rados.0.trial169.stdout:835: finishing write tid 2 to trial16941936-50 2026-02-25T08:45:33.492 INFO:tasks.rados.rados.0.trial169.stdout:835: finishing write tid 3 to trial16941936-50 2026-02-25T08:45:33.492 INFO:tasks.rados.rados.0.trial169.stdout:835: finishing write tid 4 to trial16941936-50 2026-02-25T08:45:33.492 INFO:tasks.rados.rados.0.trial169.stdout:835: oid 50 updating version 0 to 455 2026-02-25T08:45:33.493 INFO:tasks.rados.rados.0.trial169.stdout:835: oid 50 version 455 is already newer than 455 2026-02-25T08:45:33.493 INFO:tasks.rados.rados.0.trial169.stdout:835: oid 50 version 455 is already newer than 453 2026-02-25T08:45:33.493 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 50 v 455 (ObjNum 398 snap 109 seq_num 398) dirty exists 2026-02-25T08:45:33.493 INFO:tasks.rados.rados.0.trial169.stdout:835: left oid 50 (ObjNum 398 snap 109 seq_num 398) 2026-02-25T08:45:33.493 INFO:tasks.rados.rados.0.trial169.stdout:836: finishing write tid 1 to trial16941936-49 2026-02-25T08:45:33.507 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:33.510 INFO:tasks.rados.rados.0.trial169.stdout:837: finishing write tid 1 to trial16941936-19 2026-02-25T08:45:33.511 INFO:tasks.rados.rados.0.trial169.stdout:839: finishing write tid 1 to trial16941936-2 2026-02-25T08:45:33.512 INFO:tasks.rados.rados.0.trial169.stdout:838: finishing write tid 1 to trial16941936-45 2026-02-25T08:45:33.515 INFO:tasks.rados.rados.0.trial169.stdout:836: finishing write tid 2 to trial16941936-49 2026-02-25T08:45:33.519 INFO:tasks.rados.rados.0.trial169.stdout:839: finishing write tid 2 to trial16941936-2 2026-02-25T08:45:33.520 INFO:tasks.rados.rados.0.trial169.stdout:840: finishing write tid 1 to trial16941936-36 2026-02-25T08:45:33.523 INFO:tasks.rados.rados.0.trial169.stdout:839: finishing write tid 3 to trial16941936-2 2026-02-25T08:45:33.524 INFO:tasks.rados.rados.0.trial169.stdout:839: finishing write tid 4 to trial16941936-2 2026-02-25T08:45:33.525 INFO:tasks.rados.rados.0.trial169.stdout:839: oid 2 updating version 0 to 72 2026-02-25T08:45:33.525 INFO:tasks.rados.rados.0.trial169.stdout:839: oid 2 version 72 is already newer than 72 2026-02-25T08:45:33.525 INFO:tasks.rados.rados.0.trial169.stdout:839: oid 2 version 72 is already newer than 72 2026-02-25T08:45:33.525 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 2 v 72 (ObjNum 402 snap 109 seq_num 402) dirty exists 2026-02-25T08:45:33.525 INFO:tasks.rados.rados.0.trial169.stdout:839: left oid 2 (ObjNum 402 snap 109 seq_num 402) 2026-02-25T08:45:33.526 INFO:tasks.rados.rados.0.trial169.stdout:836: finishing write tid 3 to trial16941936-49 2026-02-25T08:45:33.527 INFO:tasks.rados.rados.0.trial169.stdout:836: finishing write tid 4 to trial16941936-49 2026-02-25T08:45:33.527 INFO:tasks.rados.rados.0.trial169.stdout:836: oid 49 updating version 0 to 360 2026-02-25T08:45:33.527 INFO:tasks.rados.rados.0.trial169.stdout:836: oid 49 version 360 is already newer than 360 2026-02-25T08:45:33.527 INFO:tasks.rados.rados.0.trial169.stdout:836: oid 49 version 360 is already newer than 360 2026-02-25T08:45:33.527 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 49 v 360 (ObjNum 399 snap 109 seq_num 399) dirty exists 2026-02-25T08:45:33.528 INFO:tasks.rados.rados.0.trial169.stdout:836: left oid 49 (ObjNum 399 snap 109 seq_num 399) 2026-02-25T08:45:33.528 INFO:tasks.rados.rados.0.trial169.stdout:837: finishing write tid 2 to trial16941936-19 2026-02-25T08:45:33.528 INFO:tasks.rados.rados.0.trial169.stdout:838: finishing write tid 2 to trial16941936-45 2026-02-25T08:45:33.531 INFO:tasks.rados.rados.0.trial169.stdout:837: finishing write tid 3 to trial16941936-19 2026-02-25T08:45:33.533 INFO:tasks.rados.rados.0.trial169.stdout:838: finishing write tid 3 to trial16941936-45 2026-02-25T08:45:33.533 INFO:tasks.rados.rados.0.trial169.stdout:840: finishing write tid 2 to trial16941936-36 2026-02-25T08:45:33.534 INFO:tasks.rados.rados.0.trial169.stdout:838: finishing write tid 4 to trial16941936-45 2026-02-25T08:45:33.535 INFO:tasks.rados.rados.0.trial169.stdout:838: oid 45 updating version 0 to 138 2026-02-25T08:45:33.535 INFO:tasks.rados.rados.0.trial169.stdout:838: oid 45 updating version 138 to 140 2026-02-25T08:45:33.535 INFO:tasks.rados.rados.0.trial169.stdout:838: oid 45 version 140 is already newer than 140 2026-02-25T08:45:33.535 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 45 v 140 (ObjNum 401 snap 109 seq_num 401) dirty exists 2026-02-25T08:45:33.535 INFO:tasks.rados.rados.0.trial169.stdout:838: left oid 45 (ObjNum 401 snap 109 seq_num 401) 2026-02-25T08:45:33.536 INFO:tasks.rados.rados.0.trial169.stdout:837: finishing write tid 4 to trial16941936-19 2026-02-25T08:45:33.536 INFO:tasks.rados.rados.0.trial169.stdout:837: oid 19 updating version 0 to 126 2026-02-25T08:45:33.536 INFO:tasks.rados.rados.0.trial169.stdout:837: oid 19 updating version 126 to 128 2026-02-25T08:45:33.536 INFO:tasks.rados.rados.0.trial169.stdout:837: oid 19 version 128 is already newer than 128 2026-02-25T08:45:33.536 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 19 v 128 (ObjNum 400 snap 109 seq_num 400) dirty exists 2026-02-25T08:45:33.536 INFO:tasks.rados.rados.0.trial169.stdout:837: left oid 19 (ObjNum 400 snap 109 seq_num 400) 2026-02-25T08:45:33.539 INFO:tasks.rados.rados.0.trial169.stdout:841: expect (ObjNum 357 snap 97 seq_num 357) 2026-02-25T08:45:33.557 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1551859 2026-02-25T08:45:33.584 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:33.581+0000 7f8406c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:33.585 INFO:tasks.rados.rados.0.trial169.stdout:835: done (7 left) 2026-02-25T08:45:33.586 INFO:tasks.rados.rados.0.trial169.stdout:836: done (6 left) 2026-02-25T08:45:33.586 INFO:tasks.rados.rados.0.trial169.stdout:837: done (5 left) 2026-02-25T08:45:33.586 INFO:tasks.rados.rados.0.trial169.stdout:838: done (4 left) 2026-02-25T08:45:33.586 INFO:tasks.rados.rados.0.trial169.stdout:839: done (3 left) 2026-02-25T08:45:33.586 INFO:tasks.rados.rados.0.trial169.stdout:841: done (2 left) 2026-02-25T08:45:33.586 INFO:tasks.rados.rados.0.trial169.stdout:842: done (1 left) 2026-02-25T08:45:33.586 INFO:tasks.rados.rados.0.trial169.stdout:843: write (excl) oid 14 current snap is 109 2026-02-25T08:45:33.586 INFO:tasks.rados.rados.0.trial169.stdout:840: finishing write tid 3 to trial16941936-36 2026-02-25T08:45:33.586 INFO:tasks.rados.rados.0.trial169.stdout:843: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2026-02-25T08:45:33.587 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:33.585+0000 7f8406c1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:33.590 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:33.590 INFO:tasks.rados.rados.0.trial169.stdout:843: writing trial16941936-14 from 406835 to 1049389 tid 1 2026-02-25T08:45:33.593 INFO:tasks.rados.rados.0.trial169.stdout:843: writing trial16941936-14 from 1532972 to 2039767 tid 2 2026-02-25T08:45:33.597 INFO:tasks.rados.rados.0.trial169.stdout:843: writing trial16941936-14 from 2772312 to 2806835 tid 3 2026-02-25T08:45:33.597 INFO:tasks.rados.rados.0.trial169.stdout:844: read oid 6 snap -1 2026-02-25T08:45:33.597 INFO:tasks.rados.rados.0.trial169.stdout:844: expect (ObjNum 288 snap 79 seq_num 288) 2026-02-25T08:45:33.597 INFO:tasks.rados.rados.0.trial169.stdout:845: read oid 32 snap -1 2026-02-25T08:45:33.597 INFO:tasks.rados.rados.0.trial169.stdout:845: expect deleted 2026-02-25T08:45:33.598 INFO:tasks.rados.rados.0.trial169.stdout:846: snap_create 2026-02-25T08:45:33.598 INFO:tasks.rados.rados.0.trial169.stdout:840: finishing write tid 4 to trial16941936-36 2026-02-25T08:45:33.598 INFO:tasks.rados.rados.0.trial169.stdout:840: oid 36 updating version 0 to 130 2026-02-25T08:45:33.598 INFO:tasks.rados.rados.0.trial169.stdout:840: oid 36 version 130 is already newer than 130 2026-02-25T08:45:33.598 INFO:tasks.rados.rados.0.trial169.stdout:840: oid 36 version 130 is already newer than 130 2026-02-25T08:45:33.598 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 36 v 130 (ObjNum 403 snap 109 seq_num 403) dirty exists 2026-02-25T08:45:33.598 INFO:tasks.rados.rados.0.trial169.stdout:840: left oid 36 (ObjNum 403 snap 109 seq_num 403) 2026-02-25T08:45:33.611 INFO:tasks.rados.rados.0.trial169.stdout:844: expect (ObjNum 288 snap 79 seq_num 288) 2026-02-25T08:45:33.611 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 364647 2026-02-25T08:45:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:33.648 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:33.657 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:33.681 INFO:tasks.rados.rados.0.trial169.stdout:843: finishing write tid 1 to trial16941936-14 2026-02-25T08:45:33.682 INFO:tasks.rados.rados.0.trial169.stdout:843: finishing write tid 2 to trial16941936-14 2026-02-25T08:45:33.682 INFO:tasks.rados.rados.0.trial169.stdout:843: finishing write tid 3 to trial16941936-14 2026-02-25T08:45:33.682 INFO:tasks.rados.rados.0.trial169.stdout:843: finishing write tid 4 to trial16941936-14 2026-02-25T08:45:33.682 INFO:tasks.rados.rados.0.trial169.stdout:843: finishing write tid 5 to trial16941936-14 2026-02-25T08:45:33.682 INFO:tasks.rados.rados.0.trial169.stdout:843: oid 14 updating version 0 to 145 2026-02-25T08:45:33.682 INFO:tasks.rados.rados.0.trial169.stdout:843: oid 14 version 145 is already newer than 145 2026-02-25T08:45:33.682 INFO:tasks.rados.rados.0.trial169.stdout:843: oid 14 version 145 is already newer than 145 2026-02-25T08:45:33.682 INFO:tasks.rados.rados.0.trial169.stdout:843: oid 14 version 145 is already newer than 142 2026-02-25T08:45:33.682 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 14 v 145 (ObjNum 405 snap 109 seq_num 405) dirty exists 2026-02-25T08:45:33.682 INFO:tasks.rados.rados.0.trial169.stdout:843: left oid 14 (ObjNum 405 snap 109 seq_num 405) 2026-02-25T08:45:34.170 INFO:tasks.rados.rados.0.trial169.stdout:840: done (4 left) 2026-02-25T08:45:34.170 INFO:tasks.rados.rados.0.trial169.stdout:843: done (3 left) 2026-02-25T08:45:34.170 INFO:tasks.rados.rados.0.trial169.stdout:844: done (2 left) 2026-02-25T08:45:34.170 INFO:tasks.rados.rados.0.trial169.stdout:845: done (1 left) 2026-02-25T08:45:34.170 INFO:tasks.rados.rados.0.trial169.stdout:846: done (0 left) 2026-02-25T08:45:34.170 INFO:tasks.rados.rados.0.trial169.stdout:847: write oid 44 current snap is 110 2026-02-25T08:45:34.171 INFO:tasks.rados.rados.0.trial169.stdout:847: seq_num 406 ranges {423642=717803,1875101=766806} 2026-02-25T08:45:34.174 INFO:tasks.rados.rados.0.trial169.stdout:847: writing trial16941936-44 from 423642 to 1141445 tid 1 2026-02-25T08:45:34.180 INFO:tasks.rados.rados.0.trial169.stdout:847: writing trial16941936-44 from 1875101 to 2641907 tid 2 2026-02-25T08:45:34.180 INFO:tasks.rados.rados.0.trial169.stdout:848: read oid 18 snap -1 2026-02-25T08:45:34.181 INFO:tasks.rados.rados.0.trial169.stdout:848: expect (ObjNum 329 snap 89 seq_num 329) 2026-02-25T08:45:34.181 INFO:tasks.rados.rados.0.trial169.stdout:849: delete oid 27 current snap is 110 2026-02-25T08:45:34.184 INFO:tasks.rados.rados.0.trial169.stdout:849: done (2 left) 2026-02-25T08:45:34.184 INFO:tasks.rados.rados.0.trial169.stdout:850: read oid 28 snap -1 2026-02-25T08:45:34.184 INFO:tasks.rados.rados.0.trial169.stdout:850: expect (ObjNum 370 snap 100 seq_num 370) 2026-02-25T08:45:34.184 INFO:tasks.rados.rados.0.trial169.stdout:851: delete oid 39 current snap is 110 2026-02-25T08:45:34.199 INFO:tasks.rados.rados.0.trial169.stdout:850: expect (ObjNum 370 snap 100 seq_num 370) 2026-02-25T08:45:34.224 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:34.254 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:34.283 INFO:tasks.rados.rados.0.trial169.stdout:848: expect (ObjNum 329 snap 89 seq_num 329) 2026-02-25T08:45:34.324 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1093707 2026-02-25T08:45:34.367 INFO:tasks.rados.rados.0.trial169.stdout:847: finishing write tid 1 to trial16941936-44 2026-02-25T08:45:34.367 INFO:tasks.rados.rados.0.trial169.stdout:847: finishing write tid 2 to trial16941936-44 2026-02-25T08:45:34.367 INFO:tasks.rados.rados.0.trial169.stdout:847: finishing write tid 3 to trial16941936-44 2026-02-25T08:45:34.367 INFO:tasks.rados.rados.0.trial169.stdout:847: finishing write tid 4 to trial16941936-44 2026-02-25T08:45:34.367 INFO:tasks.rados.rados.0.trial169.stdout:847: oid 44 updating version 0 to 133 2026-02-25T08:45:34.367 INFO:tasks.rados.rados.0.trial169.stdout:847: oid 44 version 133 is already newer than 133 2026-02-25T08:45:34.367 INFO:tasks.rados.rados.0.trial169.stdout:847: oid 44 version 133 is already newer than 133 2026-02-25T08:45:34.368 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 44 v 133 (ObjNum 406 snap 110 seq_num 406) dirty exists 2026-02-25T08:45:34.368 INFO:tasks.rados.rados.0.trial169.stdout:847: left oid 44 (ObjNum 406 snap 110 seq_num 406) 2026-02-25T08:45:34.368 INFO:tasks.rados.rados.0.trial169.stdout:847: done (3 left) 2026-02-25T08:45:34.368 INFO:tasks.rados.rados.0.trial169.stdout:848: done (2 left) 2026-02-25T08:45:34.368 INFO:tasks.rados.rados.0.trial169.stdout:850: done (1 left) 2026-02-25T08:45:34.368 INFO:tasks.rados.rados.0.trial169.stdout:851: done (0 left) 2026-02-25T08:45:34.368 INFO:tasks.rados.rados.0.trial169.stdout:852: read oid 45 snap -1 2026-02-25T08:45:34.368 INFO:tasks.rados.rados.0.trial169.stdout:852: expect (ObjNum 401 snap 109 seq_num 401) 2026-02-25T08:45:34.368 INFO:tasks.rados.rados.0.trial169.stdout:853: read oid 14 snap -1 2026-02-25T08:45:34.368 INFO:tasks.rados.rados.0.trial169.stdout:853: expect (ObjNum 405 snap 109 seq_num 405) 2026-02-25T08:45:34.368 INFO:tasks.rados.rados.0.trial169.stdout:854: write oid 48 current snap is 110 2026-02-25T08:45:34.368 INFO:tasks.rados.rados.0.trial169.stdout:854: seq_num 409 ranges {474063=543550,1650547=514133,2765651=108412} 2026-02-25T08:45:34.370 INFO:tasks.rados.rados.0.trial169.stdout:854: writing trial16941936-48 from 474063 to 1017613 tid 1 2026-02-25T08:45:34.375 INFO:tasks.rados.rados.0.trial169.stdout:854: writing trial16941936-48 from 1650547 to 2164680 tid 2 2026-02-25T08:45:34.378 INFO:tasks.rados.rados.0.trial169.stdout:854: writing trial16941936-48 from 2765651 to 2874063 tid 3 2026-02-25T08:45:34.378 INFO:tasks.rados.rados.0.trial169.stdout:855: read oid 33 snap -1 2026-02-25T08:45:34.378 INFO:tasks.rados.rados.0.trial169.stdout:855: expect (ObjNum 396 snap 109 seq_num 396) 2026-02-25T08:45:34.378 INFO:tasks.rados.rados.0.trial169.stdout:856: write oid 19 current snap is 110 2026-02-25T08:45:34.378 INFO:tasks.rados.rados.0.trial169.stdout:856: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2026-02-25T08:45:34.382 INFO:tasks.rados.rados.0.trial169.stdout:856: writing trial16941936-19 from 490870 to 1109669 tid 1 2026-02-25T08:45:34.386 INFO:tasks.rados.rados.0.trial169.stdout:856: writing trial16941936-19 from 1592676 to 2366820 tid 2 2026-02-25T08:45:34.388 INFO:tasks.rados.rados.0.trial169.stdout:856: writing trial16941936-19 from 2793074 to 2890870 tid 3 2026-02-25T08:45:34.389 INFO:tasks.rados.rados.0.trial169.stdout:857: snap_create 2026-02-25T08:45:34.390 INFO:tasks.rados.rados.0.trial169.stdout:852: expect (ObjNum 401 snap 109 seq_num 401) 2026-02-25T08:45:34.402 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1641637 2026-02-25T08:45:34.429 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:34.497 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2362218 2026-02-25T08:45:34.510 INFO:tasks.rados.rados.0.trial169.stdout:853: expect (ObjNum 405 snap 109 seq_num 405) 2026-02-25T08:45:34.532 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1848650 2026-02-25T08:45:34.555 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:34.607 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:34.659 INFO:tasks.rados.rados.0.trial169.stdout:855: expect (ObjNum 396 snap 109 seq_num 396) 2026-02-25T08:45:34.659 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:34.770 INFO:tasks.rados.rados.0.trial169.stdout:854: finishing write tid 1 to trial16941936-48 2026-02-25T08:45:34.770 INFO:tasks.rados.rados.0.trial169.stdout:854: finishing write tid 2 to trial16941936-48 2026-02-25T08:45:34.770 INFO:tasks.rados.rados.0.trial169.stdout:854: finishing write tid 3 to trial16941936-48 2026-02-25T08:45:34.770 INFO:tasks.rados.rados.0.trial169.stdout:854: finishing write tid 4 to trial16941936-48 2026-02-25T08:45:34.771 INFO:tasks.rados.rados.0.trial169.stdout:854: finishing write tid 5 to trial16941936-48 2026-02-25T08:45:34.771 INFO:tasks.rados.rados.0.trial169.stdout:854: oid 48 updating version 0 to 33 2026-02-25T08:45:34.771 INFO:tasks.rados.rados.0.trial169.stdout:854: oid 48 version 33 is already newer than 33 2026-02-25T08:45:34.771 INFO:tasks.rados.rados.0.trial169.stdout:854: oid 48 version 33 is already newer than 33 2026-02-25T08:45:34.771 INFO:tasks.rados.rados.0.trial169.stdout:854: oid 48 version 33 is already newer than 33 2026-02-25T08:45:34.771 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 48 v 33 (ObjNum 409 snap 110 seq_num 409) dirty exists 2026-02-25T08:45:34.771 INFO:tasks.rados.rados.0.trial169.stdout:854: left oid 48 (ObjNum 409 snap 110 seq_num 409) 2026-02-25T08:45:34.771 INFO:tasks.rados.rados.0.trial169.stdout:856: finishing write tid 1 to trial16941936-19 2026-02-25T08:45:34.771 INFO:tasks.rados.rados.0.trial169.stdout:856: finishing write tid 2 to trial16941936-19 2026-02-25T08:45:34.771 INFO:tasks.rados.rados.0.trial169.stdout:856: finishing write tid 3 to trial16941936-19 2026-02-25T08:45:34.771 INFO:tasks.rados.rados.0.trial169.stdout:856: finishing write tid 4 to trial16941936-19 2026-02-25T08:45:34.771 INFO:tasks.rados.rados.0.trial169.stdout:856: finishing write tid 5 to trial16941936-19 2026-02-25T08:45:34.772 INFO:tasks.rados.rados.0.trial169.stdout:856: oid 19 updating version 0 to 133 2026-02-25T08:45:34.772 INFO:tasks.rados.rados.0.trial169.stdout:856: oid 19 version 133 is already newer than 133 2026-02-25T08:45:34.772 INFO:tasks.rados.rados.0.trial169.stdout:856: oid 19 version 133 is already newer than 133 2026-02-25T08:45:34.772 INFO:tasks.rados.rados.0.trial169.stdout:856: oid 19 version 133 is already newer than 133 2026-02-25T08:45:34.772 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 19 v 133 (ObjNum 410 snap 110 seq_num 410) dirty exists 2026-02-25T08:45:34.772 INFO:tasks.rados.rados.0.trial169.stdout:856: left oid 19 (ObjNum 410 snap 110 seq_num 410) 2026-02-25T08:45:35.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~209s 2026-02-25T08:45:35.174 INFO:tasks.rados.rados.0.trial169.stdout:852: done (5 left) 2026-02-25T08:45:35.175 INFO:tasks.rados.rados.0.trial169.stdout:853: done (4 left) 2026-02-25T08:45:35.175 INFO:tasks.rados.rados.0.trial169.stdout:854: done (3 left) 2026-02-25T08:45:35.175 INFO:tasks.rados.rados.0.trial169.stdout:855: done (2 left) 2026-02-25T08:45:35.175 INFO:tasks.rados.rados.0.trial169.stdout:856: done (1 left) 2026-02-25T08:45:35.175 INFO:tasks.rados.rados.0.trial169.stdout:857: done (0 left) 2026-02-25T08:45:35.175 INFO:tasks.rados.rados.0.trial169.stdout:858: snap_remove snap 110 2026-02-25T08:45:35.543 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:45:35.616 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:35.619 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:35.619 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:45:35.649 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:35.724 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:35.722+0000 7fc9ea70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:35.727 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:35.725+0000 7fc9ea70b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:35.730 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:35.771 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:35.772 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:36.175 INFO:tasks.rados.rados.0.trial169.stdout:858: done (0 left) 2026-02-25T08:45:36.176 INFO:tasks.rados.rados.0.trial169.stdout:859: write (excl) oid 32 current snap is 111 2026-02-25T08:45:36.176 INFO:tasks.rados.rados.0.trial169.stdout:859: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2026-02-25T08:45:36.178 INFO:tasks.rados.rados.0.trial169.stdout:859: writing trial16941936-32 from 507677 to 918078 tid 1 2026-02-25T08:45:36.182 INFO:tasks.rados.rados.0.trial169.stdout:859: writing trial16941936-32 from 1367511 to 1885313 tid 2 2026-02-25T08:45:36.187 INFO:tasks.rados.rados.0.trial169.stdout:859: writing trial16941936-32 from 2420497 to 2907677 tid 3 2026-02-25T08:45:36.187 INFO:tasks.rados.rados.0.trial169.stdout:860: snap_remove snap 108 2026-02-25T08:45:37.181 INFO:tasks.rados.rados.0.trial169.stdout:860: done (1 left) 2026-02-25T08:45:37.181 INFO:tasks.rados.rados.0.trial169.stdout:861: snap_remove snap 102 2026-02-25T08:45:37.772 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:37.850 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:37.847+0000 7f6af156f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:37.852 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:37.851+0000 7f6af156f640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:37.856 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:37.897 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:37.897 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:38.153 INFO:tasks.rados.rados.0.trial169.stdout:861: done (1 left) 2026-02-25T08:45:38.154 INFO:tasks.rados.rados.0.trial169.stdout:862: delete oid 37 current snap is 111 2026-02-25T08:45:38.154 INFO:tasks.rados.rados.0.trial169.stdout:859: finishing write tid 1 to trial16941936-32 2026-02-25T08:45:38.154 INFO:tasks.rados.rados.0.trial169.stdout:859: finishing write tid 2 to trial16941936-32 2026-02-25T08:45:38.154 INFO:tasks.rados.rados.0.trial169.stdout:859: finishing write tid 3 to trial16941936-32 2026-02-25T08:45:38.154 INFO:tasks.rados.rados.0.trial169.stdout:859: finishing write tid 4 to trial16941936-32 2026-02-25T08:45:38.154 INFO:tasks.rados.rados.0.trial169.stdout:859: finishing write tid 5 to trial16941936-32 2026-02-25T08:45:38.154 INFO:tasks.rados.rados.0.trial169.stdout:859: oid 32 updating version 0 to 76 2026-02-25T08:45:38.154 INFO:tasks.rados.rados.0.trial169.stdout:859: oid 32 version 76 is already newer than 76 2026-02-25T08:45:38.154 INFO:tasks.rados.rados.0.trial169.stdout:859: oid 32 version 76 is already newer than 73 2026-02-25T08:45:38.154 INFO:tasks.rados.rados.0.trial169.stdout:859: oid 32 version 76 is already newer than 76 2026-02-25T08:45:38.155 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 32 v 76 (ObjNum 411 snap 111 seq_num 411) dirty exists 2026-02-25T08:45:38.155 INFO:tasks.rados.rados.0.trial169.stdout:859: left oid 32 (ObjNum 411 snap 111 seq_num 411) 2026-02-25T08:45:38.202 INFO:tasks.rados.rados.0.trial169.stdout:859: done (1 left) 2026-02-25T08:45:38.203 INFO:tasks.rados.rados.0.trial169.stdout:862: done (0 left) 2026-02-25T08:45:38.203 INFO:tasks.rados.rados.0.trial169.stdout:863: write oid 25 current snap is 111 2026-02-25T08:45:38.203 INFO:tasks.rados.rados.0.trial169.stdout:863: seq_num 413 ranges {541291=560899,1884475=521471} 2026-02-25T08:45:38.207 INFO:tasks.rados.rados.0.trial169.stdout:863: writing trial16941936-25 from 541291 to 1102190 tid 1 2026-02-25T08:45:38.212 INFO:tasks.rados.rados.0.trial169.stdout:863: writing trial16941936-25 from 1884475 to 2405946 tid 2 2026-02-25T08:45:38.212 INFO:tasks.rados.rados.0.trial169.stdout:864: read oid 48 snap -1 2026-02-25T08:45:38.213 INFO:tasks.rados.rados.0.trial169.stdout:864: expect (ObjNum 409 snap 110 seq_num 409) 2026-02-25T08:45:38.213 INFO:tasks.rados.rados.0.trial169.stdout:865: snap_create 2026-02-25T08:45:38.240 INFO:tasks.rados.rados.0.trial169.stdout:863: finishing write tid 1 to trial16941936-25 2026-02-25T08:45:38.246 INFO:tasks.rados.rados.0.trial169.stdout:863: finishing write tid 2 to trial16941936-25 2026-02-25T08:45:38.250 INFO:tasks.rados.rados.0.trial169.stdout:863: finishing write tid 3 to trial16941936-25 2026-02-25T08:45:38.252 INFO:tasks.rados.rados.0.trial169.stdout:863: finishing write tid 4 to trial16941936-25 2026-02-25T08:45:38.252 INFO:tasks.rados.rados.0.trial169.stdout:863: oid 25 updating version 0 to 89 2026-02-25T08:45:38.252 INFO:tasks.rados.rados.0.trial169.stdout:863: oid 25 version 89 is already newer than 89 2026-02-25T08:45:38.252 INFO:tasks.rados.rados.0.trial169.stdout:863: oid 25 version 89 is already newer than 89 2026-02-25T08:45:38.252 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 25 v 89 (ObjNum 413 snap 111 seq_num 413) dirty exists 2026-02-25T08:45:38.252 INFO:tasks.rados.rados.0.trial169.stdout:863: left oid 25 (ObjNum 413 snap 111 seq_num 413) 2026-02-25T08:45:38.254 INFO:tasks.rados.rados.0.trial169.stdout:864: expect (ObjNum 409 snap 110 seq_num 409) 2026-02-25T08:45:38.268 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1329891 2026-02-25T08:45:39.186 INFO:tasks.rados.rados.0.trial169.stdout:863: done (2 left) 2026-02-25T08:45:39.186 INFO:tasks.rados.rados.0.trial169.stdout:864: done (1 left) 2026-02-25T08:45:39.186 INFO:tasks.rados.rados.0.trial169.stdout:865: done (0 left) 2026-02-25T08:45:39.186 INFO:tasks.rados.rados.0.trial169.stdout:866: read oid 31 snap 95 2026-02-25T08:45:39.186 INFO:tasks.rados.rados.0.trial169.stdout:866: expect (ObjNum 308 snap 84 seq_num 308) 2026-02-25T08:45:39.186 INFO:tasks.rados.rados.0.trial169.stdout:867: write (excl) oid 21 current snap is 112 2026-02-25T08:45:39.186 INFO:tasks.rados.rados.0.trial169.stdout:867: seq_num 414 ranges {558098=636148,1826604=665129} 2026-02-25T08:45:39.190 INFO:tasks.rados.rados.0.trial169.stdout:867: writing trial16941936-21 from 558098 to 1194246 tid 1 2026-02-25T08:45:39.195 INFO:tasks.rados.rados.0.trial169.stdout:867: writing trial16941936-21 from 1826604 to 2491733 tid 2 2026-02-25T08:45:39.195 INFO:tasks.rados.rados.0.trial169.stdout:868: snap_remove snap 98 2026-02-25T08:45:39.898 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:39.976 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:39.973+0000 7f6f01d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:39.979 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:39.977+0000 7f6f01d0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:39.982 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:40.022 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:40.189 INFO:tasks.rados.rados.0.trial169.stdout:868: done (2 left) 2026-02-25T08:45:40.189 INFO:tasks.rados.rados.0.trial169.stdout:869: read oid 40 snap -1 2026-02-25T08:45:40.190 INFO:tasks.rados.rados.0.trial169.stdout:869: expect (ObjNum 383 snap 104 seq_num 383) 2026-02-25T08:45:40.190 INFO:tasks.rados.rados.0.trial169.stdout:870: read oid 33 snap -1 2026-02-25T08:45:40.190 INFO:tasks.rados.rados.0.trial169.stdout:870: expect (ObjNum 396 snap 109 seq_num 396) 2026-02-25T08:45:40.190 INFO:tasks.rados.rados.0.trial169.stdout:871: snap_remove snap 103 2026-02-25T08:45:40.620 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:45:40.666 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~215s 2026-02-25T08:45:40.693 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:40.696 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:40.696 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:45:41.192 INFO:tasks.rados.rados.0.trial169.stdout:871: done (4 left) 2026-02-25T08:45:41.192 INFO:tasks.rados.rados.0.trial169.stdout:872: snap_create 2026-02-25T08:45:41.192 INFO:tasks.rados.rados.0.trial169.stdout:866: expect (ObjNum 308 snap 84 seq_num 308) 2026-02-25T08:45:41.192 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:41.268 INFO:tasks.rados.rados.0.trial169.stdout:867: finishing write tid 1 to trial16941936-21 2026-02-25T08:45:41.268 INFO:tasks.rados.rados.0.trial169.stdout:867: finishing write tid 2 to trial16941936-21 2026-02-25T08:45:41.268 INFO:tasks.rados.rados.0.trial169.stdout:867: finishing write tid 3 to trial16941936-21 2026-02-25T08:45:41.269 INFO:tasks.rados.rados.0.trial169.stdout:867: finishing write tid 4 to trial16941936-21 2026-02-25T08:45:41.269 INFO:tasks.rados.rados.0.trial169.stdout:867: oid 21 updating version 0 to 77 2026-02-25T08:45:41.269 INFO:tasks.rados.rados.0.trial169.stdout:867: oid 21 updating version 77 to 79 2026-02-25T08:45:41.269 INFO:tasks.rados.rados.0.trial169.stdout:867: oid 21 version 79 is already newer than 79 2026-02-25T08:45:41.269 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 21 v 79 (ObjNum 414 snap 112 seq_num 414) dirty exists 2026-02-25T08:45:41.269 INFO:tasks.rados.rados.0.trial169.stdout:867: left oid 21 (ObjNum 414 snap 112 seq_num 414) 2026-02-25T08:45:41.269 INFO:tasks.rados.rados.0.trial169.stdout:869: expect (ObjNum 383 snap 104 seq_num 383) 2026-02-25T08:45:41.281 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:41.322 INFO:tasks.rados.rados.0.trial169.stdout:870: expect (ObjNum 396 snap 109 seq_num 396) 2026-02-25T08:45:41.361 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 35383 2026-02-25T08:45:42.023 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:42.100 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:42.097+0000 7fbd3565a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:42.102 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:42.101+0000 7fbd3565a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:42.106 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:42.145 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:42.196 INFO:tasks.rados.rados.0.trial169.stdout:866: done (4 left) 2026-02-25T08:45:42.196 INFO:tasks.rados.rados.0.trial169.stdout:867: done (3 left) 2026-02-25T08:45:42.196 INFO:tasks.rados.rados.0.trial169.stdout:869: done (2 left) 2026-02-25T08:45:42.196 INFO:tasks.rados.rados.0.trial169.stdout:870: done (1 left) 2026-02-25T08:45:42.197 INFO:tasks.rados.rados.0.trial169.stdout:872: done (0 left) 2026-02-25T08:45:42.197 INFO:tasks.rados.rados.0.trial169.stdout:873: write oid 29 current snap is 113 2026-02-25T08:45:42.197 INFO:tasks.rados.rados.0.trial169.stdout:873: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2026-02-25T08:45:42.201 INFO:tasks.rados.rados.0.trial169.stdout:873: writing trial16941936-29 from 574905 to 1286302 tid 1 2026-02-25T08:45:42.206 INFO:tasks.rados.rados.0.trial169.stdout:873: writing trial16941936-29 from 1885086 to 2410226 tid 2 2026-02-25T08:45:42.209 INFO:tasks.rados.rados.0.trial169.stdout:873: writing trial16941936-29 from 2930189 to 2974905 tid 3 2026-02-25T08:45:42.209 INFO:tasks.rados.rados.0.trial169.stdout:874: read oid 2 snap -1 2026-02-25T08:45:42.209 INFO:tasks.rados.rados.0.trial169.stdout:874: expect (ObjNum 402 snap 109 seq_num 402) 2026-02-25T08:45:42.209 INFO:tasks.rados.rados.0.trial169.stdout:875: read oid 46 snap -1 2026-02-25T08:45:42.209 INFO:tasks.rados.rados.0.trial169.stdout:875: expect (ObjNum 388 snap 106 seq_num 388) 2026-02-25T08:45:42.209 INFO:tasks.rados.rados.0.trial169.stdout:876: read oid 30 snap -1 2026-02-25T08:45:42.209 INFO:tasks.rados.rados.0.trial169.stdout:876: expect (ObjNum 339 snap 94 seq_num 339) 2026-02-25T08:45:42.209 INFO:tasks.rados.rados.0.trial169.stdout:877: delete oid 9 current snap is 113 2026-02-25T08:45:42.231 INFO:tasks.rados.rados.0.trial169.stdout:874: expect (ObjNum 402 snap 109 seq_num 402) 2026-02-25T08:45:42.283 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2423790 2026-02-25T08:45:42.285 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:42.321 INFO:tasks.rados.rados.0.trial169.stdout:874: done (4 left) 2026-02-25T08:45:42.321 INFO:tasks.rados.rados.0.trial169.stdout:877: done (3 left) 2026-02-25T08:45:42.321 INFO:tasks.rados.rados.0.trial169.stdout:878: snap_remove snap 95 2026-02-25T08:45:43.198 INFO:tasks.rados.rados.0.trial169.stdout:878: done (3 left) 2026-02-25T08:45:43.198 INFO:tasks.rados.rados.0.trial169.stdout:879: write oid 9 current snap is 113 2026-02-25T08:45:43.198 INFO:tasks.rados.rados.0.trial169.stdout:875: expect (ObjNum 388 snap 106 seq_num 388) 2026-02-25T08:45:43.208 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:43.250 INFO:tasks.rados.rados.0.trial169.stdout:876: expect (ObjNum 339 snap 94 seq_num 339) 2026-02-25T08:45:43.250 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:43.281 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:43.311 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:43.342 INFO:tasks.rados.rados.0.trial169.stdout:873: finishing write tid 1 to trial16941936-29 2026-02-25T08:45:43.342 INFO:tasks.rados.rados.0.trial169.stdout:873: finishing write tid 2 to trial16941936-29 2026-02-25T08:45:43.342 INFO:tasks.rados.rados.0.trial169.stdout:879: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2026-02-25T08:45:43.346 INFO:tasks.rados.rados.0.trial169.stdout:879: writing trial16941936-9 from 608519 to 1070414 tid 1 2026-02-25T08:45:43.351 INFO:tasks.rados.rados.0.trial169.stdout:879: writing trial16941936-9 from 1602050 to 2130859 tid 2 2026-02-25T08:45:43.356 INFO:tasks.rados.rados.0.trial169.stdout:879: writing trial16941936-9 from 2585035 to 3008519 tid 3 2026-02-25T08:45:43.356 INFO:tasks.rados.rados.0.trial169.stdout:875: done (3 left) 2026-02-25T08:45:43.356 INFO:tasks.rados.rados.0.trial169.stdout:876: done (2 left) 2026-02-25T08:45:43.356 INFO:tasks.rados.rados.0.trial169.stdout:880: read oid 8 snap 93 2026-02-25T08:45:43.357 INFO:tasks.rados.rados.0.trial169.stdout:880: expect deleted 2026-02-25T08:45:43.357 INFO:tasks.rados.rados.0.trial169.stdout:881: write oid 35 current snap is 113 2026-02-25T08:45:43.357 INFO:tasks.rados.rados.0.trial169.stdout:873: finishing write tid 3 to trial16941936-29 2026-02-25T08:45:43.357 INFO:tasks.rados.rados.0.trial169.stdout:873: finishing write tid 4 to trial16941936-29 2026-02-25T08:45:43.357 INFO:tasks.rados.rados.0.trial169.stdout:873: finishing write tid 5 to trial16941936-29 2026-02-25T08:45:43.357 INFO:tasks.rados.rados.0.trial169.stdout:873: oid 29 updating version 0 to 470 2026-02-25T08:45:43.357 INFO:tasks.rados.rados.0.trial169.stdout:873: oid 29 version 470 is already newer than 470 2026-02-25T08:45:43.357 INFO:tasks.rados.rados.0.trial169.stdout:873: oid 29 version 470 is already newer than 470 2026-02-25T08:45:43.357 INFO:tasks.rados.rados.0.trial169.stdout:873: oid 29 version 470 is already newer than 470 2026-02-25T08:45:43.357 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 29 v 470 (ObjNum 415 snap 113 seq_num 415) dirty exists 2026-02-25T08:45:43.357 INFO:tasks.rados.rados.0.trial169.stdout:873: left oid 29 (ObjNum 415 snap 113 seq_num 415) 2026-02-25T08:45:43.358 INFO:tasks.rados.rados.0.trial169.stdout:881: seq_num 418 ranges {625326=537144,1944179=672467} 2026-02-25T08:45:43.360 INFO:tasks.rados.rados.0.trial169.stdout:881: writing trial16941936-35 from 625326 to 1162470 tid 1 2026-02-25T08:45:43.366 INFO:tasks.rados.rados.0.trial169.stdout:881: writing trial16941936-35 from 1944179 to 2616646 tid 2 2026-02-25T08:45:43.366 INFO:tasks.rados.rados.0.trial169.stdout:873: done (3 left) 2026-02-25T08:45:43.366 INFO:tasks.rados.rados.0.trial169.stdout:882: delete oid 40 current snap is 113 2026-02-25T08:45:43.372 INFO:tasks.rados.rados.0.trial169.stdout:879: finishing write tid 1 to trial16941936-9 2026-02-25T08:45:43.396 INFO:tasks.rados.rados.0.trial169.stdout:879: finishing write tid 2 to trial16941936-9 2026-02-25T08:45:43.405 INFO:tasks.rados.rados.0.trial169.stdout:879: finishing write tid 3 to trial16941936-9 2026-02-25T08:45:43.411 INFO:tasks.rados.rados.0.trial169.stdout:879: finishing write tid 4 to trial16941936-9 2026-02-25T08:45:43.412 INFO:tasks.rados.rados.0.trial169.stdout:881: finishing write tid 1 to trial16941936-35 2026-02-25T08:45:43.413 INFO:tasks.rados.rados.0.trial169.stdout:879: finishing write tid 5 to trial16941936-9 2026-02-25T08:45:43.413 INFO:tasks.rados.rados.0.trial169.stdout:879: oid 9 updating version 0 to 139 2026-02-25T08:45:43.413 INFO:tasks.rados.rados.0.trial169.stdout:879: oid 9 version 139 is already newer than 139 2026-02-25T08:45:43.413 INFO:tasks.rados.rados.0.trial169.stdout:879: oid 9 version 139 is already newer than 139 2026-02-25T08:45:43.413 INFO:tasks.rados.rados.0.trial169.stdout:879: oid 9 version 139 is already newer than 139 2026-02-25T08:45:43.414 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 9 v 139 (ObjNum 417 snap 113 seq_num 417) dirty exists 2026-02-25T08:45:43.414 INFO:tasks.rados.rados.0.trial169.stdout:879: left oid 9 (ObjNum 417 snap 113 seq_num 417) 2026-02-25T08:45:43.418 INFO:tasks.rados.rados.0.trial169.stdout:881: finishing write tid 2 to trial16941936-35 2026-02-25T08:45:43.424 INFO:tasks.rados.rados.0.trial169.stdout:881: finishing write tid 3 to trial16941936-35 2026-02-25T08:45:43.427 INFO:tasks.rados.rados.0.trial169.stdout:881: finishing write tid 4 to trial16941936-35 2026-02-25T08:45:43.427 INFO:tasks.rados.rados.0.trial169.stdout:881: oid 35 updating version 0 to 478 2026-02-25T08:45:43.427 INFO:tasks.rados.rados.0.trial169.stdout:881: oid 35 version 478 is already newer than 478 2026-02-25T08:45:43.427 INFO:tasks.rados.rados.0.trial169.stdout:881: oid 35 version 478 is already newer than 478 2026-02-25T08:45:43.427 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 35 v 478 (ObjNum 418 snap 113 seq_num 418) dirty exists 2026-02-25T08:45:43.428 INFO:tasks.rados.rados.0.trial169.stdout:881: left oid 35 (ObjNum 418 snap 113 seq_num 418) 2026-02-25T08:45:43.438 INFO:tasks.rados.rados.0.trial169.stdout:879: done (3 left) 2026-02-25T08:45:43.438 INFO:tasks.rados.rados.0.trial169.stdout:880: done (2 left) 2026-02-25T08:45:43.438 INFO:tasks.rados.rados.0.trial169.stdout:881: done (1 left) 2026-02-25T08:45:43.438 INFO:tasks.rados.rados.0.trial169.stdout:882: done (0 left) 2026-02-25T08:45:43.438 INFO:tasks.rados.rados.0.trial169.stdout:883: write (excl) oid 24 current snap is 113 2026-02-25T08:45:43.438 INFO:tasks.rados.rados.0.trial169.stdout:883: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2026-02-25T08:45:43.442 INFO:tasks.rados.rados.0.trial169.stdout:883: writing trial16941936-24 from 658940 to 1062935 tid 1 2026-02-25T08:45:43.447 INFO:tasks.rados.rados.0.trial169.stdout:883: writing trial16941936-24 from 1777496 to 2453632 tid 2 2026-02-25T08:45:43.450 INFO:tasks.rados.rados.0.trial169.stdout:883: writing trial16941936-24 from 2950951 to 3058940 tid 3 2026-02-25T08:45:43.450 INFO:tasks.rados.rados.0.trial169.stdout:884: read oid 8 snap -1 2026-02-25T08:45:43.450 INFO:tasks.rados.rados.0.trial169.stdout:884: expect deleted 2026-02-25T08:45:43.450 INFO:tasks.rados.rados.0.trial169.stdout:885: delete oid 32 current snap is 113 2026-02-25T08:45:43.462 INFO:tasks.rados.rados.0.trial169.stdout:883: finishing write tid 1 to trial16941936-24 2026-02-25T08:45:43.478 INFO:tasks.rados.rados.0.trial169.stdout:883: finishing write tid 2 to trial16941936-24 2026-02-25T08:45:43.485 INFO:tasks.rados.rados.0.trial169.stdout:883: finishing write tid 3 to trial16941936-24 2026-02-25T08:45:43.491 INFO:tasks.rados.rados.0.trial169.stdout:883: finishing write tid 4 to trial16941936-24 2026-02-25T08:45:43.495 INFO:tasks.rados.rados.0.trial169.stdout:883: finishing write tid 5 to trial16941936-24 2026-02-25T08:45:43.495 INFO:tasks.rados.rados.0.trial169.stdout:883: oid 24 updating version 0 to 78 2026-02-25T08:45:43.495 INFO:tasks.rados.rados.0.trial169.stdout:883: oid 24 version 78 is already newer than 78 2026-02-25T08:45:43.495 INFO:tasks.rados.rados.0.trial169.stdout:883: oid 24 version 78 is already newer than 78 2026-02-25T08:45:43.495 INFO:tasks.rados.rados.0.trial169.stdout:883: oid 24 version 78 is already newer than 75 2026-02-25T08:45:43.495 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 24 v 78 (ObjNum 420 snap 113 seq_num 420) dirty exists 2026-02-25T08:45:43.495 INFO:tasks.rados.rados.0.trial169.stdout:883: left oid 24 (ObjNum 420 snap 113 seq_num 420) 2026-02-25T08:45:43.501 INFO:tasks.rados.rados.0.trial169.stdout:883: done (2 left) 2026-02-25T08:45:43.501 INFO:tasks.rados.rados.0.trial169.stdout:884: done (1 left) 2026-02-25T08:45:43.501 INFO:tasks.rados.rados.0.trial169.stdout:885: done (0 left) 2026-02-25T08:45:43.501 INFO:tasks.rados.rados.0.trial169.stdout:886: read oid 21 snap 109 2026-02-25T08:45:43.501 INFO:tasks.rados.rados.0.trial169.stdout:886: expect deleted 2026-02-25T08:45:43.501 INFO:tasks.rados.rados.0.trial169.stdout:887: snap_remove snap 105 2026-02-25T08:45:44.146 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:44.202 INFO:tasks.rados.rados.0.trial169.stdout:887: done (1 left) 2026-02-25T08:45:44.202 INFO:tasks.rados.rados.0.trial169.stdout:888: read oid 36 snap -1 2026-02-25T08:45:44.202 INFO:tasks.rados.rados.0.trial169.stdout:888: expect (ObjNum 403 snap 109 seq_num 403) 2026-02-25T08:45:44.202 INFO:tasks.rados.rados.0.trial169.stdout:889: write oid 26 current snap is 113 2026-02-25T08:45:44.202 INFO:tasks.rados.rados.0.trial169.stdout:889: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2026-02-25T08:45:44.206 INFO:tasks.rados.rados.0.trial169.stdout:889: writing trial16941936-26 from 692554 to 1247047 tid 1 2026-02-25T08:45:44.211 INFO:tasks.rados.rados.0.trial169.stdout:889: writing trial16941936-26 from 1778107 to 2457912 tid 2 2026-02-25T08:45:44.214 INFO:tasks.rados.rados.0.trial169.stdout:889: writing trial16941936-26 from 2889444 to 3092554 tid 3 2026-02-25T08:45:44.214 INFO:tasks.rados.rados.0.trial169.stdout:890: write oid 15 current snap is 113 2026-02-25T08:45:44.214 INFO:tasks.rados.rados.0.trial169.stdout:890: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2026-02-25T08:45:44.218 INFO:tasks.rados.rados.0.trial169.stdout:890: writing trial16941936-15 from 709361 to 1339103 tid 1 2026-02-25T08:45:44.222 INFO:tasks.rados.rados.0.trial169.stdout:890: writing trial16941936-15 from 1836589 to 2376405 tid 2 2026-02-25T08:45:44.226 INFO:tasks.rados.rados.0.trial169.stdout:890: writing trial16941936-15 from 3033220 to 3109361 tid 3 2026-02-25T08:45:44.226 INFO:tasks.rados.rados.0.trial169.stdout:891: read oid 41 snap -1 2026-02-25T08:45:44.226 INFO:tasks.rados.rados.0.trial169.stdout:891: expect (ObjNum 338 snap 93 seq_num 338) 2026-02-25T08:45:44.226 INFO:tasks.rados.rados.0.trial169.stdout:892: read oid 42 snap -1 2026-02-25T08:45:44.226 INFO:tasks.rados.rados.0.trial169.stdout:892: expect (ObjNum 324 snap 87 seq_num 324) 2026-02-25T08:45:44.226 INFO:tasks.rados.rados.0.trial169.stdout:893: delete oid 6 current snap is 113 2026-02-25T08:45:44.227 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:44.224+0000 7f466ec5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:44.229 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:44.228+0000 7f466ec5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:44.233 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:44.249 INFO:tasks.rados.rados.0.trial169.stdout:892: expect (ObjNum 324 snap 87 seq_num 324) 2026-02-25T08:45:44.252 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 617831 2026-02-25T08:45:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:44.303 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:44.328 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:44.368 INFO:tasks.rados.rados.0.trial169.stdout:888: expect (ObjNum 403 snap 109 seq_num 403) 2026-02-25T08:45:44.368 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:44.392 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:44.440 INFO:tasks.rados.rados.0.trial169.stdout:891: expect (ObjNum 338 snap 93 seq_num 338) 2026-02-25T08:45:44.467 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:44.529 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2922203 2026-02-25T08:45:44.539 INFO:tasks.rados.rados.0.trial169.stdout:890: finishing write tid 1 to trial16941936-15 2026-02-25T08:45:44.539 INFO:tasks.rados.rados.0.trial169.stdout:886: done (6 left) 2026-02-25T08:45:44.539 INFO:tasks.rados.rados.0.trial169.stdout:888: done (5 left) 2026-02-25T08:45:44.539 INFO:tasks.rados.rados.0.trial169.stdout:891: done (4 left) 2026-02-25T08:45:44.539 INFO:tasks.rados.rados.0.trial169.stdout:892: done (3 left) 2026-02-25T08:45:44.539 INFO:tasks.rados.rados.0.trial169.stdout:893: done (2 left) 2026-02-25T08:45:44.540 INFO:tasks.rados.rados.0.trial169.stdout:894: snap_remove snap 112 2026-02-25T08:45:45.206 INFO:tasks.rados.rados.0.trial169.stdout:894: done (2 left) 2026-02-25T08:45:45.207 INFO:tasks.rados.rados.0.trial169.stdout:895: delete oid 23 current snap is 113 2026-02-25T08:45:45.207 INFO:tasks.rados.rados.0.trial169.stdout:890: finishing write tid 2 to trial16941936-15 2026-02-25T08:45:45.207 INFO:tasks.rados.rados.0.trial169.stdout:890: finishing write tid 3 to trial16941936-15 2026-02-25T08:45:45.207 INFO:tasks.rados.rados.0.trial169.stdout:890: finishing write tid 4 to trial16941936-15 2026-02-25T08:45:45.207 INFO:tasks.rados.rados.0.trial169.stdout:890: finishing write tid 5 to trial16941936-15 2026-02-25T08:45:45.207 INFO:tasks.rados.rados.0.trial169.stdout:890: oid 15 updating version 0 to 145 2026-02-25T08:45:45.207 INFO:tasks.rados.rados.0.trial169.stdout:890: oid 15 version 145 is already newer than 145 2026-02-25T08:45:45.207 INFO:tasks.rados.rados.0.trial169.stdout:890: oid 15 version 145 is already newer than 145 2026-02-25T08:45:45.207 INFO:tasks.rados.rados.0.trial169.stdout:890: oid 15 version 145 is already newer than 145 2026-02-25T08:45:45.207 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 15 v 145 (ObjNum 423 snap 113 seq_num 423) dirty exists 2026-02-25T08:45:45.207 INFO:tasks.rados.rados.0.trial169.stdout:890: left oid 15 (ObjNum 423 snap 113 seq_num 423) 2026-02-25T08:45:45.208 INFO:tasks.rados.rados.0.trial169.stdout:889: finishing write tid 1 to trial16941936-26 2026-02-25T08:45:45.208 INFO:tasks.rados.rados.0.trial169.stdout:889: finishing write tid 2 to trial16941936-26 2026-02-25T08:45:45.208 INFO:tasks.rados.rados.0.trial169.stdout:889: finishing write tid 3 to trial16941936-26 2026-02-25T08:45:45.208 INFO:tasks.rados.rados.0.trial169.stdout:889: finishing write tid 4 to trial16941936-26 2026-02-25T08:45:45.208 INFO:tasks.rados.rados.0.trial169.stdout:889: finishing write tid 5 to trial16941936-26 2026-02-25T08:45:45.208 INFO:tasks.rados.rados.0.trial169.stdout:889: oid 26 updating version 0 to 383 2026-02-25T08:45:45.208 INFO:tasks.rados.rados.0.trial169.stdout:889: oid 26 version 383 is already newer than 383 2026-02-25T08:45:45.208 INFO:tasks.rados.rados.0.trial169.stdout:889: oid 26 version 383 is already newer than 383 2026-02-25T08:45:45.208 INFO:tasks.rados.rados.0.trial169.stdout:889: oid 26 version 383 is already newer than 383 2026-02-25T08:45:45.208 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 26 v 383 (ObjNum 422 snap 113 seq_num 422) dirty exists 2026-02-25T08:45:45.208 INFO:tasks.rados.rados.0.trial169.stdout:889: left oid 26 (ObjNum 422 snap 113 seq_num 422) 2026-02-25T08:45:45.244 INFO:tasks.rados.rados.0.trial169.stdout:889: done (2 left) 2026-02-25T08:45:45.244 INFO:tasks.rados.rados.0.trial169.stdout:890: done (1 left) 2026-02-25T08:45:45.245 INFO:tasks.rados.rados.0.trial169.stdout:895: done (0 left) 2026-02-25T08:45:45.245 INFO:tasks.rados.rados.0.trial169.stdout:896: read oid 16 snap -1 2026-02-25T08:45:45.245 INFO:tasks.rados.rados.0.trial169.stdout:896: expect (ObjNum 249 snap 71 seq_num 249) 2026-02-25T08:45:45.245 INFO:tasks.rados.rados.0.trial169.stdout:897: write oid 41 current snap is 113 2026-02-25T08:45:45.245 INFO:tasks.rados.rados.0.trial169.stdout:897: seq_num 426 ranges {759782=571842,2012035=687143} 2026-02-25T08:45:45.249 INFO:tasks.rados.rados.0.trial169.stdout:897: writing trial16941936-41 from 759782 to 1331624 tid 1 2026-02-25T08:45:45.254 INFO:tasks.rados.rados.0.trial169.stdout:897: writing trial16941936-41 from 2012035 to 2699178 tid 2 2026-02-25T08:45:45.255 INFO:tasks.rados.rados.0.trial169.stdout:898: delete oid 20 current snap is 113 2026-02-25T08:45:45.270 INFO:tasks.rados.rados.0.trial169.stdout:896: expect (ObjNum 249 snap 71 seq_num 249) 2026-02-25T08:45:45.270 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 551847 2026-02-25T08:45:45.315 INFO:tasks.rados.rados.0.trial169.stdout:897: finishing write tid 1 to trial16941936-41 2026-02-25T08:45:45.315 INFO:tasks.rados.rados.0.trial169.stdout:897: finishing write tid 2 to trial16941936-41 2026-02-25T08:45:45.315 INFO:tasks.rados.rados.0.trial169.stdout:897: finishing write tid 3 to trial16941936-41 2026-02-25T08:45:45.315 INFO:tasks.rados.rados.0.trial169.stdout:897: finishing write tid 4 to trial16941936-41 2026-02-25T08:45:45.315 INFO:tasks.rados.rados.0.trial169.stdout:897: oid 41 updating version 0 to 387 2026-02-25T08:45:45.315 INFO:tasks.rados.rados.0.trial169.stdout:897: oid 41 version 387 is already newer than 387 2026-02-25T08:45:45.315 INFO:tasks.rados.rados.0.trial169.stdout:897: oid 41 version 387 is already newer than 387 2026-02-25T08:45:45.315 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 41 v 387 (ObjNum 426 snap 113 seq_num 426) dirty exists 2026-02-25T08:45:45.316 INFO:tasks.rados.rados.0.trial169.stdout:897: left oid 41 (ObjNum 426 snap 113 seq_num 426) 2026-02-25T08:45:45.321 INFO:tasks.rados.rados.0.trial169.stdout:896: done (2 left) 2026-02-25T08:45:45.321 INFO:tasks.rados.rados.0.trial169.stdout:897: done (1 left) 2026-02-25T08:45:45.322 INFO:tasks.rados.rados.0.trial169.stdout:898: done (0 left) 2026-02-25T08:45:45.322 INFO:tasks.rados.rados.0.trial169.stdout:899: read oid 3 snap 109 2026-02-25T08:45:45.322 INFO:tasks.rados.rados.0.trial169.stdout:899: expect (ObjNum 387 snap 104 seq_num 387) 2026-02-25T08:45:45.322 INFO:tasks.rados.rados.0.trial169.stdout:900: snap_remove snap 93 2026-02-25T08:45:45.697 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:45:45.769 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:45.772 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:45.773 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:45:46.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~220s 2026-02-25T08:45:46.210 INFO:tasks.rados.rados.0.trial169.stdout:900: done (1 left) 2026-02-25T08:45:46.210 INFO:tasks.rados.rados.0.trial169.stdout:901: snap_remove snap 111 2026-02-25T08:45:46.305 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:46.383 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:46.380+0000 7fa3ee10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:46.385 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:46.383+0000 7fa3ee10b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:46.388 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:46.427 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:46.427 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:47.216 INFO:tasks.rados.rados.0.trial169.stdout:901: done (1 left) 2026-02-25T08:45:47.216 INFO:tasks.rados.rados.0.trial169.stdout:902: snap_create 2026-02-25T08:45:47.216 INFO:tasks.rados.rados.0.trial169.stdout:899: expect (ObjNum 387 snap 104 seq_num 387) 2026-02-25T08:45:48.155 INFO:tasks.rados.rados.0.trial169.stdout:899: done (1 left) 2026-02-25T08:45:48.156 INFO:tasks.rados.rados.0.trial169.stdout:902: done (0 left) 2026-02-25T08:45:48.156 INFO:tasks.rados.rados.0.trial169.stdout:903: snap_remove snap 113 2026-02-25T08:45:48.428 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:48.504 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:48.501+0000 7fea5ddad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:48.506 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:48.505+0000 7fea5ddad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:48.510 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:48.546 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:48.547 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:49.157 INFO:tasks.rados.rados.0.trial169.stdout:903: done (0 left) 2026-02-25T08:45:49.157 INFO:tasks.rados.rados.0.trial169.stdout:904: snap_create 2026-02-25T08:45:50.157 INFO:tasks.rados.rados.0.trial169.stdout:904: done (0 left) 2026-02-25T08:45:50.158 INFO:tasks.rados.rados.0.trial169.stdout:905: snap_remove snap 114 2026-02-25T08:45:50.548 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:50.625 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:50.622+0000 7fb988fa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:50.628 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:50.626+0000 7fb988fa4640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:50.632 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:50.670 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:50.671 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:50.774 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:45:50.846 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:50.849 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:50.849 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:45:51.159 INFO:tasks.rados.rados.0.trial169.stdout:905: done (0 left) 2026-02-25T08:45:51.159 INFO:tasks.rados.rados.0.trial169.stdout:906: read oid 22 snap -1 2026-02-25T08:45:51.159 INFO:tasks.rados.rados.0.trial169.stdout:906: expect (ObjNum 282 snap 76 seq_num 282) 2026-02-25T08:45:51.160 INFO:tasks.rados.rados.0.trial169.stdout:907: write oid 3 current snap is 115 2026-02-25T08:45:51.160 INFO:tasks.rados.rados.0.trial169.stdout:907: seq_num 428 ranges {793396=722340,2128999=690812} 2026-02-25T08:45:51.165 INFO:tasks.rados.rados.0.trial169.stdout:907: writing trial16941936-3 from 793396 to 1515736 tid 1 2026-02-25T08:45:51.170 INFO:tasks.rados.rados.0.trial169.stdout:907: writing trial16941936-3 from 2128999 to 2819811 tid 2 2026-02-25T08:45:51.171 INFO:tasks.rados.rados.0.trial169.stdout:908: write oid 34 current snap is 115 2026-02-25T08:45:51.171 INFO:tasks.rados.rados.0.trial169.stdout:908: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2026-02-25T08:45:51.175 INFO:tasks.rados.rados.0.trial169.stdout:908: writing trial16941936-34 from 410203 to 1207792 tid 1 2026-02-25T08:45:51.178 INFO:tasks.rados.rados.0.trial169.stdout:908: writing trial16941936-34 from 1787481 to 2221951 tid 2 2026-02-25T08:45:51.183 INFO:tasks.rados.rados.0.trial169.stdout:908: writing trial16941936-34 from 2681405 to 3210203 tid 3 2026-02-25T08:45:51.183 INFO:tasks.rados.rados.0.trial169.stdout:909: read oid 43 snap 109 2026-02-25T08:45:51.183 INFO:tasks.rados.rados.0.trial169.stdout:909: expect (ObjNum 386 snap 104 seq_num 386) 2026-02-25T08:45:51.183 INFO:tasks.rados.rados.0.trial169.stdout:910: delete oid 4 current snap is 115 2026-02-25T08:45:51.201 INFO:tasks.rados.rados.0.trial169.stdout:906: expect (ObjNum 282 snap 76 seq_num 282) 2026-02-25T08:45:51.201 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:51.224 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:51.247 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:51.270 INFO:tasks.rados.rados.0.trial169.stdout:906: done (4 left) 2026-02-25T08:45:51.271 INFO:tasks.rados.rados.0.trial169.stdout:910: done (3 left) 2026-02-25T08:45:51.271 INFO:tasks.rados.rados.0.trial169.stdout:911: read oid 47 snap -1 2026-02-25T08:45:51.271 INFO:tasks.rados.rados.0.trial169.stdout:911: expect (ObjNum 371 snap 100 seq_num 371) 2026-02-25T08:45:51.271 INFO:tasks.rados.rados.0.trial169.stdout:912: delete oid 27 current snap is 115 2026-02-25T08:45:51.271 INFO:tasks.rados.rados.0.trial169.stdout:909: expect (ObjNum 386 snap 104 seq_num 386) 2026-02-25T08:45:51.271 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:51.309 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:51.385 INFO:tasks.rados.rados.0.trial169.stdout:907: finishing write tid 1 to trial16941936-3 2026-02-25T08:45:51.385 INFO:tasks.rados.rados.0.trial169.stdout:909: done (4 left) 2026-02-25T08:45:51.385 INFO:tasks.rados.rados.0.trial169.stdout:912: done (3 left) 2026-02-25T08:45:51.386 INFO:tasks.rados.rados.0.trial169.stdout:913: read oid 43 snap -1 2026-02-25T08:45:51.386 INFO:tasks.rados.rados.0.trial169.stdout:913: expect (ObjNum 386 snap 104 seq_num 386) 2026-02-25T08:45:51.386 INFO:tasks.rados.rados.0.trial169.stdout:914: write (excl) oid 17 current snap is 115 2026-02-25T08:45:51.386 INFO:tasks.rados.rados.0.trial169.stdout:907: finishing write tid 2 to trial16941936-3 2026-02-25T08:45:51.386 INFO:tasks.rados.rados.0.trial169.stdout:908: finishing write tid 1 to trial16941936-34 2026-02-25T08:45:51.386 INFO:tasks.rados.rados.0.trial169.stdout:907: finishing write tid 3 to trial16941936-3 2026-02-25T08:45:51.386 INFO:tasks.rados.rados.0.trial169.stdout:914: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2026-02-25T08:45:51.389 INFO:tasks.rados.rados.0.trial169.stdout:914: writing trial16941936-17 from 460624 to 1083960 tid 1 2026-02-25T08:45:51.394 INFO:tasks.rados.rados.0.trial169.stdout:914: writing trial16941936-17 from 1846574 to 2544724 tid 2 2026-02-25T08:45:51.397 INFO:tasks.rados.rados.0.trial169.stdout:914: writing trial16941936-17 from 3163674 to 3260624 tid 3 2026-02-25T08:45:51.397 INFO:tasks.rados.rados.0.trial169.stdout:915: snap_remove snap 109 2026-02-25T08:45:51.674 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~226s 2026-02-25T08:45:52.165 INFO:tasks.rados.rados.0.trial169.stdout:915: done (5 left) 2026-02-25T08:45:52.165 INFO:tasks.rados.rados.0.trial169.stdout:916: write oid 22 current snap is 115 2026-02-25T08:45:52.165 INFO:tasks.rados.rados.0.trial169.stdout:907: finishing write tid 4 to trial16941936-3 2026-02-25T08:45:52.165 INFO:tasks.rados.rados.0.trial169.stdout:907: oid 3 updating version 0 to 199 2026-02-25T08:45:52.165 INFO:tasks.rados.rados.0.trial169.stdout:907: oid 3 version 199 is already newer than 199 2026-02-25T08:45:52.165 INFO:tasks.rados.rados.0.trial169.stdout:907: oid 3 version 199 is already newer than 199 2026-02-25T08:45:52.165 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 3 v 199 (ObjNum 428 snap 115 seq_num 428) dirty exists 2026-02-25T08:45:52.165 INFO:tasks.rados.rados.0.trial169.stdout:907: left oid 3 (ObjNum 428 snap 115 seq_num 428) 2026-02-25T08:45:52.165 INFO:tasks.rados.rados.0.trial169.stdout:916: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2026-02-25T08:45:52.169 INFO:tasks.rados.rados.0.trial169.stdout:916: writing trial16941936-22 from 477431 to 1176016 tid 1 2026-02-25T08:45:52.173 INFO:tasks.rados.rados.0.trial169.stdout:916: writing trial16941936-22 from 1905056 to 2346864 tid 2 2026-02-25T08:45:52.177 INFO:tasks.rados.rados.0.trial169.stdout:916: writing trial16941936-22 from 3074744 to 3277431 tid 3 2026-02-25T08:45:52.177 INFO:tasks.rados.rados.0.trial169.stdout:907: done (5 left) 2026-02-25T08:45:52.177 INFO:tasks.rados.rados.0.trial169.stdout:917: write (excl) oid 14 current snap is 115 2026-02-25T08:45:52.177 INFO:tasks.rados.rados.0.trial169.stdout:908: finishing write tid 2 to trial16941936-34 2026-02-25T08:45:52.177 INFO:tasks.rados.rados.0.trial169.stdout:908: finishing write tid 3 to trial16941936-34 2026-02-25T08:45:52.177 INFO:tasks.rados.rados.0.trial169.stdout:908: finishing write tid 4 to trial16941936-34 2026-02-25T08:45:52.177 INFO:tasks.rados.rados.0.trial169.stdout:917: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2026-02-25T08:45:52.180 INFO:tasks.rados.rados.0.trial169.stdout:917: writing trial16941936-14 from 494238 to 984425 tid 1 2026-02-25T08:45:52.185 INFO:tasks.rados.rados.0.trial169.stdout:917: writing trial16941936-14 from 1563538 to 2265357 tid 2 2026-02-25T08:45:52.190 INFO:tasks.rados.rados.0.trial169.stdout:917: writing trial16941936-14 from 2818520 to 3294238 tid 3 2026-02-25T08:45:52.190 INFO:tasks.rados.rados.0.trial169.stdout:918: read oid 32 snap -1 2026-02-25T08:45:52.190 INFO:tasks.rados.rados.0.trial169.stdout:918: expect deleted 2026-02-25T08:45:52.190 INFO:tasks.rados.rados.0.trial169.stdout:919: delete oid 6 current snap is 115 2026-02-25T08:45:52.190 INFO:tasks.rados.rados.0.trial169.stdout:908: finishing write tid 5 to trial16941936-34 2026-02-25T08:45:52.190 INFO:tasks.rados.rados.0.trial169.stdout:908: oid 34 updating version 0 to 146 2026-02-25T08:45:52.190 INFO:tasks.rados.rados.0.trial169.stdout:908: oid 34 version 146 is already newer than 146 2026-02-25T08:45:52.190 INFO:tasks.rados.rados.0.trial169.stdout:908: oid 34 version 146 is already newer than 146 2026-02-25T08:45:52.190 INFO:tasks.rados.rados.0.trial169.stdout:908: oid 34 version 146 is already newer than 146 2026-02-25T08:45:52.191 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 34 v 146 (ObjNum 429 snap 115 seq_num 429) dirty exists 2026-02-25T08:45:52.191 INFO:tasks.rados.rados.0.trial169.stdout:908: left oid 34 (ObjNum 429 snap 115 seq_num 429) 2026-02-25T08:45:52.191 INFO:tasks.rados.rados.0.trial169.stdout:911: expect (ObjNum 371 snap 100 seq_num 371) 2026-02-25T08:45:52.191 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:52.242 INFO:tasks.rados.rados.0.trial169.stdout:913: expect (ObjNum 386 snap 104 seq_num 386) 2026-02-25T08:45:52.242 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:52.319 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:52.360 INFO:tasks.rados.rados.0.trial169.stdout:914: finishing write tid 1 to trial16941936-17 2026-02-25T08:45:52.360 INFO:tasks.rados.rados.0.trial169.stdout:914: finishing write tid 2 to trial16941936-17 2026-02-25T08:45:52.360 INFO:tasks.rados.rados.0.trial169.stdout:914: finishing write tid 3 to trial16941936-17 2026-02-25T08:45:52.360 INFO:tasks.rados.rados.0.trial169.stdout:914: finishing write tid 4 to trial16941936-17 2026-02-25T08:45:52.360 INFO:tasks.rados.rados.0.trial169.stdout:914: finishing write tid 5 to trial16941936-17 2026-02-25T08:45:52.360 INFO:tasks.rados.rados.0.trial169.stdout:914: oid 17 updating version 0 to 83 2026-02-25T08:45:52.360 INFO:tasks.rados.rados.0.trial169.stdout:914: oid 17 updating version 83 to 86 2026-02-25T08:45:52.361 INFO:tasks.rados.rados.0.trial169.stdout:914: oid 17 version 86 is already newer than 86 2026-02-25T08:45:52.361 INFO:tasks.rados.rados.0.trial169.stdout:914: oid 17 version 86 is already newer than 86 2026-02-25T08:45:52.361 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 17 v 86 (ObjNum 432 snap 115 seq_num 432) dirty exists 2026-02-25T08:45:52.361 INFO:tasks.rados.rados.0.trial169.stdout:914: left oid 17 (ObjNum 432 snap 115 seq_num 432) 2026-02-25T08:45:52.361 INFO:tasks.rados.rados.0.trial169.stdout:908: done (7 left) 2026-02-25T08:45:52.361 INFO:tasks.rados.rados.0.trial169.stdout:911: done (6 left) 2026-02-25T08:45:52.361 INFO:tasks.rados.rados.0.trial169.stdout:913: done (5 left) 2026-02-25T08:45:52.361 INFO:tasks.rados.rados.0.trial169.stdout:914: done (4 left) 2026-02-25T08:45:52.361 INFO:tasks.rados.rados.0.trial169.stdout:919: done (3 left) 2026-02-25T08:45:52.361 INFO:tasks.rados.rados.0.trial169.stdout:920: snap_create 2026-02-25T08:45:52.361 INFO:tasks.rados.rados.0.trial169.stdout:917: finishing write tid 1 to trial16941936-14 2026-02-25T08:45:52.361 INFO:tasks.rados.rados.0.trial169.stdout:916: finishing write tid 1 to trial16941936-22 2026-02-25T08:45:52.362 INFO:tasks.rados.rados.0.trial169.stdout:917: finishing write tid 2 to trial16941936-14 2026-02-25T08:45:52.362 INFO:tasks.rados.rados.0.trial169.stdout:916: finishing write tid 2 to trial16941936-22 2026-02-25T08:45:52.362 INFO:tasks.rados.rados.0.trial169.stdout:917: finishing write tid 3 to trial16941936-14 2026-02-25T08:45:52.362 INFO:tasks.rados.rados.0.trial169.stdout:917: finishing write tid 4 to trial16941936-14 2026-02-25T08:45:52.362 INFO:tasks.rados.rados.0.trial169.stdout:917: finishing write tid 5 to trial16941936-14 2026-02-25T08:45:52.362 INFO:tasks.rados.rados.0.trial169.stdout:917: oid 14 updating version 0 to 148 2026-02-25T08:45:52.362 INFO:tasks.rados.rados.0.trial169.stdout:917: oid 14 updating version 148 to 151 2026-02-25T08:45:52.362 INFO:tasks.rados.rados.0.trial169.stdout:917: oid 14 version 151 is already newer than 151 2026-02-25T08:45:52.362 INFO:tasks.rados.rados.0.trial169.stdout:917: oid 14 version 151 is already newer than 151 2026-02-25T08:45:52.362 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 14 v 151 (ObjNum 434 snap 115 seq_num 434) dirty exists 2026-02-25T08:45:52.362 INFO:tasks.rados.rados.0.trial169.stdout:917: left oid 14 (ObjNum 434 snap 115 seq_num 434) 2026-02-25T08:45:52.362 INFO:tasks.rados.rados.0.trial169.stdout:916: finishing write tid 3 to trial16941936-22 2026-02-25T08:45:52.363 INFO:tasks.rados.rados.0.trial169.stdout:916: finishing write tid 4 to trial16941936-22 2026-02-25T08:45:52.363 INFO:tasks.rados.rados.0.trial169.stdout:916: finishing write tid 5 to trial16941936-22 2026-02-25T08:45:52.363 INFO:tasks.rados.rados.0.trial169.stdout:916: oid 22 updating version 0 to 157 2026-02-25T08:45:52.363 INFO:tasks.rados.rados.0.trial169.stdout:916: oid 22 version 157 is already newer than 157 2026-02-25T08:45:52.363 INFO:tasks.rados.rados.0.trial169.stdout:916: oid 22 version 157 is already newer than 157 2026-02-25T08:45:52.363 INFO:tasks.rados.rados.0.trial169.stdout:916: oid 22 version 157 is already newer than 157 2026-02-25T08:45:52.363 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 22 v 157 (ObjNum 433 snap 115 seq_num 433) dirty exists 2026-02-25T08:45:52.363 INFO:tasks.rados.rados.0.trial169.stdout:916: left oid 22 (ObjNum 433 snap 115 seq_num 433) 2026-02-25T08:45:52.672 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:52.751 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:52.748+0000 7fe6c1cb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:52.753 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:52.752+0000 7fe6c1cb7640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:52.757 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:52.802 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:52.802 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:53.167 INFO:tasks.rados.rados.0.trial169.stdout:916: done (3 left) 2026-02-25T08:45:53.167 INFO:tasks.rados.rados.0.trial169.stdout:917: done (2 left) 2026-02-25T08:45:53.167 INFO:tasks.rados.rados.0.trial169.stdout:918: done (1 left) 2026-02-25T08:45:53.167 INFO:tasks.rados.rados.0.trial169.stdout:920: done (0 left) 2026-02-25T08:45:53.167 INFO:tasks.rados.rados.0.trial169.stdout:921: snap_remove snap 115 2026-02-25T08:45:54.170 INFO:tasks.rados.rados.0.trial169.stdout:921: done (0 left) 2026-02-25T08:45:54.170 INFO:tasks.rados.rados.0.trial169.stdout:922: write (excl) oid 43 current snap is 116 2026-02-25T08:45:54.170 INFO:tasks.rados.rados.0.trial169.stdout:922: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2026-02-25T08:45:54.174 INFO:tasks.rados.rados.0.trial169.stdout:922: writing trial16941936-43 from 527852 to 1168537 tid 1 2026-02-25T08:45:54.178 INFO:tasks.rados.rados.0.trial169.stdout:922: writing trial16941936-43 from 1680502 to 2269637 tid 2 2026-02-25T08:45:54.182 INFO:tasks.rados.rados.0.trial169.stdout:922: writing trial16941936-43 from 2757013 to 3327852 tid 3 2026-02-25T08:45:54.182 INFO:tasks.rados.rados.0.trial169.stdout:923: delete oid 28 current snap is 116 2026-02-25T08:45:54.198 INFO:tasks.rados.rados.0.trial169.stdout:922: finishing write tid 1 to trial16941936-43 2026-02-25T08:45:54.201 INFO:tasks.rados.rados.0.trial169.stdout:923: done (1 left) 2026-02-25T08:45:54.201 INFO:tasks.rados.rados.0.trial169.stdout:924: delete oid 29 current snap is 116 2026-02-25T08:45:54.216 INFO:tasks.rados.rados.0.trial169.stdout:922: finishing write tid 2 to trial16941936-43 2026-02-25T08:45:54.220 INFO:tasks.rados.rados.0.trial169.stdout:924: done (1 left) 2026-02-25T08:45:54.221 INFO:tasks.rados.rados.0.trial169.stdout:925: read oid 13 snap -1 2026-02-25T08:45:54.221 INFO:tasks.rados.rados.0.trial169.stdout:925: expect deleted 2026-02-25T08:45:54.221 INFO:tasks.rados.rados.0.trial169.stdout:926: delete oid 7 current snap is 116 2026-02-25T08:45:54.221 INFO:tasks.rados.rados.0.trial169.stdout:922: finishing write tid 3 to trial16941936-43 2026-02-25T08:45:54.224 INFO:tasks.rados.rados.0.trial169.stdout:922: finishing write tid 4 to trial16941936-43 2026-02-25T08:45:54.226 INFO:tasks.rados.rados.0.trial169.stdout:925: done (2 left) 2026-02-25T08:45:54.226 INFO:tasks.rados.rados.0.trial169.stdout:926: done (1 left) 2026-02-25T08:45:54.226 INFO:tasks.rados.rados.0.trial169.stdout:927: write (excl) oid 31 current snap is 116 2026-02-25T08:45:54.227 INFO:tasks.rados.rados.0.trial169.stdout:927: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2026-02-25T08:45:54.230 INFO:tasks.rados.rados.0.trial169.stdout:927: writing trial16941936-31 from 595080 to 1136761 tid 1 2026-02-25T08:45:54.235 INFO:tasks.rados.rados.0.trial169.stdout:927: writing trial16941936-31 from 1798077 to 2394550 tid 2 2026-02-25T08:45:54.239 INFO:tasks.rados.rados.0.trial169.stdout:927: writing trial16941936-31 from 3150352 to 3395080 tid 3 2026-02-25T08:45:54.239 INFO:tasks.rados.rados.0.trial169.stdout:928: snap_create 2026-02-25T08:45:54.239 INFO:tasks.rados.rados.0.trial169.stdout:922: finishing write tid 5 to trial16941936-43 2026-02-25T08:45:54.239 INFO:tasks.rados.rados.0.trial169.stdout:922: oid 43 updating version 0 to 87 2026-02-25T08:45:54.240 INFO:tasks.rados.rados.0.trial169.stdout:922: oid 43 version 87 is already newer than 84 2026-02-25T08:45:54.240 INFO:tasks.rados.rados.0.trial169.stdout:922: oid 43 version 87 is already newer than 87 2026-02-25T08:45:54.240 INFO:tasks.rados.rados.0.trial169.stdout:922: oid 43 version 87 is already newer than 87 2026-02-25T08:45:54.240 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 43 v 87 (ObjNum 436 snap 116 seq_num 436) dirty exists 2026-02-25T08:45:54.240 INFO:tasks.rados.rados.0.trial169.stdout:922: left oid 43 (ObjNum 436 snap 116 seq_num 436) 2026-02-25T08:45:54.244 INFO:tasks.rados.rados.0.trial169.stdout:927: finishing write tid 1 to trial16941936-31 2026-02-25T08:45:54.260 INFO:tasks.rados.rados.0.trial169.stdout:927: finishing write tid 2 to trial16941936-31 2026-02-25T08:45:54.266 INFO:tasks.rados.rados.0.trial169.stdout:927: finishing write tid 3 to trial16941936-31 2026-02-25T08:45:54.270 INFO:tasks.rados.rados.0.trial169.stdout:927: finishing write tid 4 to trial16941936-31 2026-02-25T08:45:54.271 INFO:tasks.rados.rados.0.trial169.stdout:927: finishing write tid 5 to trial16941936-31 2026-02-25T08:45:54.271 INFO:tasks.rados.rados.0.trial169.stdout:927: oid 31 updating version 0 to 407 2026-02-25T08:45:54.271 INFO:tasks.rados.rados.0.trial169.stdout:927: oid 31 version 407 is already newer than 404 2026-02-25T08:45:54.272 INFO:tasks.rados.rados.0.trial169.stdout:927: oid 31 version 407 is already newer than 407 2026-02-25T08:45:54.272 INFO:tasks.rados.rados.0.trial169.stdout:927: oid 31 version 407 is already newer than 407 2026-02-25T08:45:54.272 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 31 v 407 (ObjNum 440 snap 116 seq_num 440) dirty exists 2026-02-25T08:45:54.272 INFO:tasks.rados.rados.0.trial169.stdout:927: left oid 31 (ObjNum 440 snap 116 seq_num 440) 2026-02-25T08:45:54.803 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:54.882 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:54.880+0000 7f7f8bc51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:54.885 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:54.883+0000 7f7f8bc51640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:54.888 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:54.930 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:55.176 INFO:tasks.rados.rados.0.trial169.stdout:922: done (2 left) 2026-02-25T08:45:55.176 INFO:tasks.rados.rados.0.trial169.stdout:927: done (1 left) 2026-02-25T08:45:55.176 INFO:tasks.rados.rados.0.trial169.stdout:928: done (0 left) 2026-02-25T08:45:55.176 INFO:tasks.rados.rados.0.trial169.stdout:929: delete oid 40 current snap is 117 2026-02-25T08:45:55.192 INFO:tasks.rados.rados.0.trial169.stdout:929: done (0 left) 2026-02-25T08:45:55.193 INFO:tasks.rados.rados.0.trial169.stdout:930: write (excl) oid 27 current snap is 117 2026-02-25T08:45:55.193 INFO:tasks.rados.rados.0.trial169.stdout:930: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2026-02-25T08:45:55.196 INFO:tasks.rados.rados.0.trial169.stdout:930: writing trial16941936-27 from 628694 to 1037226 tid 1 2026-02-25T08:45:55.201 INFO:tasks.rados.rados.0.trial169.stdout:930: writing trial16941936-27 from 1515041 to 2115183 tid 2 2026-02-25T08:45:55.205 INFO:tasks.rados.rados.0.trial169.stdout:930: writing trial16941936-27 from 2805198 to 3357310 tid 3 2026-02-25T08:45:55.206 INFO:tasks.rados.rados.0.trial169.stdout:931: delete oid 49 current snap is 117 2026-02-25T08:45:55.217 INFO:tasks.rados.rados.0.trial169.stdout:930: finishing write tid 1 to trial16941936-27 2026-02-25T08:45:55.231 INFO:tasks.rados.rados.0.trial169.stdout:930: finishing write tid 2 to trial16941936-27 2026-02-25T08:45:55.238 INFO:tasks.rados.rados.0.trial169.stdout:930: finishing write tid 3 to trial16941936-27 2026-02-25T08:45:55.245 INFO:tasks.rados.rados.0.trial169.stdout:930: finishing write tid 4 to trial16941936-27 2026-02-25T08:45:55.246 INFO:tasks.rados.rados.0.trial169.stdout:930: finishing write tid 5 to trial16941936-27 2026-02-25T08:45:55.246 INFO:tasks.rados.rados.0.trial169.stdout:930: oid 27 updating version 0 to 91 2026-02-25T08:45:55.246 INFO:tasks.rados.rados.0.trial169.stdout:930: oid 27 version 91 is already newer than 88 2026-02-25T08:45:55.247 INFO:tasks.rados.rados.0.trial169.stdout:930: oid 27 version 91 is already newer than 91 2026-02-25T08:45:55.247 INFO:tasks.rados.rados.0.trial169.stdout:930: oid 27 version 91 is already newer than 91 2026-02-25T08:45:55.247 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 27 v 91 (ObjNum 442 snap 117 seq_num 442) dirty exists 2026-02-25T08:45:55.247 INFO:tasks.rados.rados.0.trial169.stdout:930: left oid 27 (ObjNum 442 snap 117 seq_num 442) 2026-02-25T08:45:55.265 INFO:tasks.rados.rados.0.trial169.stdout:930: done (1 left) 2026-02-25T08:45:55.265 INFO:tasks.rados.rados.0.trial169.stdout:931: done (0 left) 2026-02-25T08:45:55.265 INFO:tasks.rados.rados.0.trial169.stdout:932: delete oid 23 current snap is 117 2026-02-25T08:45:55.267 INFO:tasks.rados.rados.0.trial169.stdout:932: done (0 left) 2026-02-25T08:45:55.267 INFO:tasks.rados.rados.0.trial169.stdout:933: snap_create 2026-02-25T08:45:55.850 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:45:55.923 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:45:55.926 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:45:55.927 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:45:56.179 INFO:tasks.rados.rados.0.trial169.stdout:933: done (0 left) 2026-02-25T08:45:56.179 INFO:tasks.rados.rados.0.trial169.stdout:934: snap_create 2026-02-25T08:45:56.932 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:45:57.009 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:57.006+0000 7f0a808cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:57.011 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:57.009+0000 7f0a808cb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:57.014 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:57.053 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:57.053 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:57.178 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~231s 2026-02-25T08:45:57.183 INFO:tasks.rados.rados.0.trial169.stdout:934: done (0 left) 2026-02-25T08:45:57.184 INFO:tasks.rados.rados.0.trial169.stdout:935: read oid 20 snap -1 2026-02-25T08:45:57.184 INFO:tasks.rados.rados.0.trial169.stdout:935: expect deleted 2026-02-25T08:45:57.184 INFO:tasks.rados.rados.0.trial169.stdout:936: read oid 29 snap -1 2026-02-25T08:45:57.184 INFO:tasks.rados.rados.0.trial169.stdout:936: expect deleted 2026-02-25T08:45:57.184 INFO:tasks.rados.rados.0.trial169.stdout:937: write (excl) oid 50 current snap is 119 2026-02-25T08:45:57.184 INFO:tasks.rados.rados.0.trial169.stdout:937: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2026-02-25T08:45:57.187 INFO:tasks.rados.rados.0.trial169.stdout:937: writing trial16941936-50 from 679115 to 1313394 tid 1 2026-02-25T08:45:57.192 INFO:tasks.rados.rados.0.trial169.stdout:937: writing trial16941936-50 from 2090487 to 2554309 tid 2 2026-02-25T08:45:57.196 INFO:tasks.rados.rados.0.trial169.stdout:937: writing trial16941936-50 from 3003820 to 3479115 tid 3 2026-02-25T08:45:57.196 INFO:tasks.rados.rados.0.trial169.stdout:938: snap_create 2026-02-25T08:45:57.229 INFO:tasks.rados.rados.0.trial169.stdout:937: finishing write tid 1 to trial16941936-50 2026-02-25T08:45:57.244 INFO:tasks.rados.rados.0.trial169.stdout:937: finishing write tid 2 to trial16941936-50 2026-02-25T08:45:57.254 INFO:tasks.rados.rados.0.trial169.stdout:937: finishing write tid 3 to trial16941936-50 2026-02-25T08:45:57.257 INFO:tasks.rados.rados.0.trial169.stdout:937: finishing write tid 4 to trial16941936-50 2026-02-25T08:45:57.259 INFO:tasks.rados.rados.0.trial169.stdout:937: finishing write tid 5 to trial16941936-50 2026-02-25T08:45:57.259 INFO:tasks.rados.rados.0.trial169.stdout:937: oid 50 updating version 0 to 508 2026-02-25T08:45:57.259 INFO:tasks.rados.rados.0.trial169.stdout:937: oid 50 version 508 is already newer than 505 2026-02-25T08:45:57.259 INFO:tasks.rados.rados.0.trial169.stdout:937: oid 50 version 508 is already newer than 508 2026-02-25T08:45:57.259 INFO:tasks.rados.rados.0.trial169.stdout:937: oid 50 version 508 is already newer than 508 2026-02-25T08:45:57.259 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 50 v 508 (ObjNum 445 snap 119 seq_num 445) dirty exists 2026-02-25T08:45:57.259 INFO:tasks.rados.rados.0.trial169.stdout:937: left oid 50 (ObjNum 445 snap 119 seq_num 445) 2026-02-25T08:45:58.157 INFO:tasks.rados.rados.0.trial169.stdout:935: done (3 left) 2026-02-25T08:45:58.157 INFO:tasks.rados.rados.0.trial169.stdout:936: done (2 left) 2026-02-25T08:45:58.158 INFO:tasks.rados.rados.0.trial169.stdout:937: done (1 left) 2026-02-25T08:45:58.158 INFO:tasks.rados.rados.0.trial169.stdout:938: done (0 left) 2026-02-25T08:45:58.158 INFO:tasks.rados.rados.0.trial169.stdout:939: delete oid 46 current snap is 120 2026-02-25T08:45:58.195 INFO:tasks.rados.rados.0.trial169.stdout:939: done (0 left) 2026-02-25T08:45:58.196 INFO:tasks.rados.rados.0.trial169.stdout:940: delete oid 16 current snap is 120 2026-02-25T08:45:58.219 INFO:tasks.rados.rados.0.trial169.stdout:940: done (0 left) 2026-02-25T08:45:58.220 INFO:tasks.rados.rados.0.trial169.stdout:941: write (excl) oid 24 current snap is 120 2026-02-25T08:45:58.220 INFO:tasks.rados.rados.0.trial169.stdout:941: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2026-02-25T08:45:58.223 INFO:tasks.rados.rados.0.trial169.stdout:941: writing trial16941936-24 from 729536 to 1189562 tid 1 2026-02-25T08:45:58.228 INFO:tasks.rados.rados.0.trial169.stdout:941: writing trial16941936-24 from 1749580 to 2360729 tid 2 2026-02-25T08:45:58.232 INFO:tasks.rados.rados.0.trial169.stdout:941: writing trial16941936-24 from 2853383 to 3305833 tid 3 2026-02-25T08:45:58.232 INFO:tasks.rados.rados.0.trial169.stdout:942: read oid 40 snap -1 2026-02-25T08:45:58.232 INFO:tasks.rados.rados.0.trial169.stdout:942: expect deleted 2026-02-25T08:45:58.232 INFO:tasks.rados.rados.0.trial169.stdout:943: snap_create 2026-02-25T08:45:58.252 INFO:tasks.rados.rados.0.trial169.stdout:941: finishing write tid 1 to trial16941936-24 2026-02-25T08:45:58.265 INFO:tasks.rados.rados.0.trial169.stdout:941: finishing write tid 2 to trial16941936-24 2026-02-25T08:45:58.270 INFO:tasks.rados.rados.0.trial169.stdout:941: finishing write tid 3 to trial16941936-24 2026-02-25T08:45:58.273 INFO:tasks.rados.rados.0.trial169.stdout:941: finishing write tid 4 to trial16941936-24 2026-02-25T08:45:58.275 INFO:tasks.rados.rados.0.trial169.stdout:941: finishing write tid 5 to trial16941936-24 2026-02-25T08:45:58.275 INFO:tasks.rados.rados.0.trial169.stdout:941: oid 24 updating version 0 to 83 2026-02-25T08:45:58.276 INFO:tasks.rados.rados.0.trial169.stdout:941: oid 24 version 83 is already newer than 83 2026-02-25T08:45:58.276 INFO:tasks.rados.rados.0.trial169.stdout:941: oid 24 version 83 is already newer than 83 2026-02-25T08:45:58.276 INFO:tasks.rados.rados.0.trial169.stdout:941: oid 24 version 83 is already newer than 80 2026-02-25T08:45:58.276 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 24 v 83 (ObjNum 448 snap 120 seq_num 448) dirty exists 2026-02-25T08:45:58.276 INFO:tasks.rados.rados.0.trial169.stdout:941: left oid 24 (ObjNum 448 snap 120 seq_num 448) 2026-02-25T08:45:59.054 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:45:59.131 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:59.129+0000 7f4cccc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:59.133 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:45:59.132+0000 7f4cccc5a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:45:59.136 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:45:59.176 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:45:59.176 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:45:59.187 INFO:tasks.rados.rados.0.trial169.stdout:941: done (2 left) 2026-02-25T08:45:59.188 INFO:tasks.rados.rados.0.trial169.stdout:942: done (1 left) 2026-02-25T08:45:59.188 INFO:tasks.rados.rados.0.trial169.stdout:943: done (0 left) 2026-02-25T08:45:59.188 INFO:tasks.rados.rados.0.trial169.stdout:944: read oid 1 snap 117 2026-02-25T08:45:59.188 INFO:tasks.rados.rados.0.trial169.stdout:944: expect (ObjNum 381 snap 102 seq_num 381) 2026-02-25T08:45:59.188 INFO:tasks.rados.rados.0.trial169.stdout:945: delete oid 9 current snap is 121 2026-02-25T08:45:59.248 INFO:tasks.rados.rados.0.trial169.stdout:944: expect (ObjNum 381 snap 102 seq_num 381) 2026-02-25T08:45:59.319 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:45:59.356 INFO:tasks.rados.rados.0.trial169.stdout:944: done (1 left) 2026-02-25T08:45:59.356 INFO:tasks.rados.rados.0.trial169.stdout:945: done (0 left) 2026-02-25T08:45:59.356 INFO:tasks.rados.rados.0.trial169.stdout:946: write (excl) oid 36 current snap is 121 2026-02-25T08:45:59.356 INFO:tasks.rados.rados.0.trial169.stdout:946: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2026-02-25T08:45:59.361 INFO:tasks.rados.rados.0.trial169.stdout:946: writing trial16941936-36 from 763150 to 1490027 tid 1 2026-02-25T08:45:59.365 INFO:tasks.rados.rados.0.trial169.stdout:946: writing trial16941936-36 from 1982897 to 2597715 tid 2 2026-02-25T08:45:59.369 INFO:tasks.rados.rados.0.trial169.stdout:946: writing trial16941936-36 from 3024582 to 3499576 tid 3 2026-02-25T08:45:59.369 INFO:tasks.rados.rados.0.trial169.stdout:947: read oid 24 snap -1 2026-02-25T08:45:59.369 INFO:tasks.rados.rados.0.trial169.stdout:947: expect (ObjNum 448 snap 120 seq_num 448) 2026-02-25T08:45:59.369 INFO:tasks.rados.rados.0.trial169.stdout:948: delete oid 28 current snap is 121 2026-02-25T08:45:59.373 INFO:tasks.rados.rados.0.trial169.stdout:948: done (2 left) 2026-02-25T08:45:59.374 INFO:tasks.rados.rados.0.trial169.stdout:949: snap_remove snap 116 2026-02-25T08:46:00.193 INFO:tasks.rados.rados.0.trial169.stdout:949: done (2 left) 2026-02-25T08:46:00.194 INFO:tasks.rados.rados.0.trial169.stdout:950: snap_create 2026-02-25T08:46:00.194 INFO:tasks.rados.rados.0.trial169.stdout:947: expect (ObjNum 448 snap 120 seq_num 448) 2026-02-25T08:46:00.271 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 3158999 2026-02-25T08:46:00.278 INFO:tasks.rados.rados.0.trial169.stdout:946: finishing write tid 1 to trial16941936-36 2026-02-25T08:46:00.278 INFO:tasks.rados.rados.0.trial169.stdout:946: finishing write tid 2 to trial16941936-36 2026-02-25T08:46:00.278 INFO:tasks.rados.rados.0.trial169.stdout:946: finishing write tid 3 to trial16941936-36 2026-02-25T08:46:00.279 INFO:tasks.rados.rados.0.trial169.stdout:946: finishing write tid 4 to trial16941936-36 2026-02-25T08:46:00.279 INFO:tasks.rados.rados.0.trial169.stdout:946: finishing write tid 5 to trial16941936-36 2026-02-25T08:46:00.279 INFO:tasks.rados.rados.0.trial169.stdout:946: oid 36 updating version 0 to 142 2026-02-25T08:46:00.279 INFO:tasks.rados.rados.0.trial169.stdout:946: oid 36 version 142 is already newer than 142 2026-02-25T08:46:00.279 INFO:tasks.rados.rados.0.trial169.stdout:946: oid 36 version 142 is already newer than 142 2026-02-25T08:46:00.279 INFO:tasks.rados.rados.0.trial169.stdout:946: oid 36 version 142 is already newer than 139 2026-02-25T08:46:00.279 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 36 v 142 (ObjNum 450 snap 121 seq_num 450) dirty exists 2026-02-25T08:46:00.279 INFO:tasks.rados.rados.0.trial169.stdout:946: left oid 36 (ObjNum 450 snap 121 seq_num 450) 2026-02-25T08:46:00.928 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:46:00.999 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:01.002 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:01.003 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:46:01.178 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:01.196 INFO:tasks.rados.rados.0.trial169.stdout:946: done (2 left) 2026-02-25T08:46:01.196 INFO:tasks.rados.rados.0.trial169.stdout:947: done (1 left) 2026-02-25T08:46:01.196 INFO:tasks.rados.rados.0.trial169.stdout:950: done (0 left) 2026-02-25T08:46:01.196 INFO:tasks.rados.rados.0.trial169.stdout:951: read oid 22 snap -1 2026-02-25T08:46:01.196 INFO:tasks.rados.rados.0.trial169.stdout:951: expect (ObjNum 433 snap 115 seq_num 433) 2026-02-25T08:46:01.196 INFO:tasks.rados.rados.0.trial169.stdout:952: write (excl) oid 5 current snap is 122 2026-02-25T08:46:01.196 INFO:tasks.rados.rados.0.trial169.stdout:952: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2026-02-25T08:46:01.200 INFO:tasks.rados.rados.0.trial169.stdout:952: writing trial16941936-5 from 796764 to 1274139 tid 1 2026-02-25T08:46:01.206 INFO:tasks.rados.rados.0.trial169.stdout:952: writing trial16941936-5 from 1983508 to 2601995 tid 2 2026-02-25T08:46:01.209 INFO:tasks.rados.rados.0.trial169.stdout:952: writing trial16941936-5 from 3363075 to 3596764 tid 3 2026-02-25T08:46:01.210 INFO:tasks.rados.rados.0.trial169.stdout:953: write (excl) oid 21 current snap is 122 2026-02-25T08:46:01.210 INFO:tasks.rados.rados.0.trial169.stdout:953: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2026-02-25T08:46:01.213 INFO:tasks.rados.rados.0.trial169.stdout:953: writing trial16941936-21 from 413571 to 966195 tid 1 2026-02-25T08:46:01.218 INFO:tasks.rados.rados.0.trial169.stdout:953: writing trial16941936-21 from 1641990 to 2404135 tid 2 2026-02-25T08:46:01.223 INFO:tasks.rados.rados.0.trial169.stdout:953: writing trial16941936-21 from 2990498 to 3415661 tid 3 2026-02-25T08:46:01.223 INFO:tasks.rados.rados.0.trial169.stdout:954: write oid 14 current snap is 122 2026-02-25T08:46:01.223 INFO:tasks.rados.rados.0.trial169.stdout:954: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2026-02-25T08:46:01.226 INFO:tasks.rados.rados.0.trial169.stdout:954: writing trial16941936-14 from 430378 to 1058251 tid 1 2026-02-25T08:46:01.231 INFO:tasks.rados.rados.0.trial169.stdout:954: writing trial16941936-14 from 1700472 to 2322628 tid 2 2026-02-25T08:46:01.236 INFO:tasks.rados.rados.0.trial169.stdout:954: writing trial16941936-14 from 3017921 to 3630378 tid 3 2026-02-25T08:46:01.237 INFO:tasks.rados.rados.0.trial169.stdout:955: write oid 23 current snap is 122 2026-02-25T08:46:01.237 INFO:tasks.rados.rados.0.trial169.stdout:955: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2026-02-25T08:46:01.241 INFO:tasks.rados.rados.0.trial169.stdout:955: writing trial16941936-23 from 447185 to 1150307 tid 1 2026-02-25T08:46:01.245 INFO:tasks.rados.rados.0.trial169.stdout:955: writing trial16941936-23 from 1642601 to 2408415 tid 2 2026-02-25T08:46:01.249 INFO:tasks.rados.rados.0.trial169.stdout:955: writing trial16941936-23 from 2928991 to 3376698 tid 3 2026-02-25T08:46:01.249 INFO:tasks.rados.rados.0.trial169.stdout:956: snap_create 2026-02-25T08:46:01.249 INFO:tasks.rados.rados.0.trial169.stdout:951: expect (ObjNum 433 snap 115 seq_num 433) 2026-02-25T08:46:01.257 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:01.255+0000 7f096c9ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:01.260 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:01.258+0000 7f096c9ec640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:01.264 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:01.320 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:01.321 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:01.326 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:01.364 INFO:tasks.rados.rados.0.trial169.stdout:953: finishing write tid 1 to trial16941936-21 2026-02-25T08:46:01.364 INFO:tasks.rados.rados.0.trial169.stdout:955: finishing write tid 1 to trial16941936-23 2026-02-25T08:46:01.365 INFO:tasks.rados.rados.0.trial169.stdout:955: finishing write tid 2 to trial16941936-23 2026-02-25T08:46:01.365 INFO:tasks.rados.rados.0.trial169.stdout:954: finishing write tid 1 to trial16941936-14 2026-02-25T08:46:01.365 INFO:tasks.rados.rados.0.trial169.stdout:953: finishing write tid 2 to trial16941936-21 2026-02-25T08:46:01.365 INFO:tasks.rados.rados.0.trial169.stdout:953: finishing write tid 3 to trial16941936-21 2026-02-25T08:46:01.365 INFO:tasks.rados.rados.0.trial169.stdout:954: finishing write tid 2 to trial16941936-14 2026-02-25T08:46:01.365 INFO:tasks.rados.rados.0.trial169.stdout:955: finishing write tid 3 to trial16941936-23 2026-02-25T08:46:01.365 INFO:tasks.rados.rados.0.trial169.stdout:955: finishing write tid 4 to trial16941936-23 2026-02-25T08:46:01.365 INFO:tasks.rados.rados.0.trial169.stdout:955: finishing write tid 5 to trial16941936-23 2026-02-25T08:46:01.365 INFO:tasks.rados.rados.0.trial169.stdout:955: oid 23 updating version 0 to 152 2026-02-25T08:46:01.365 INFO:tasks.rados.rados.0.trial169.stdout:955: oid 23 version 152 is already newer than 152 2026-02-25T08:46:01.365 INFO:tasks.rados.rados.0.trial169.stdout:955: oid 23 version 152 is already newer than 152 2026-02-25T08:46:01.365 INFO:tasks.rados.rados.0.trial169.stdout:955: oid 23 version 152 is already newer than 152 2026-02-25T08:46:01.365 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 23 v 152 (ObjNum 455 snap 122 seq_num 455) dirty exists 2026-02-25T08:46:01.366 INFO:tasks.rados.rados.0.trial169.stdout:955: left oid 23 (ObjNum 455 snap 122 seq_num 455) 2026-02-25T08:46:01.366 INFO:tasks.rados.rados.0.trial169.stdout:953: finishing write tid 4 to trial16941936-21 2026-02-25T08:46:01.366 INFO:tasks.rados.rados.0.trial169.stdout:954: finishing write tid 3 to trial16941936-14 2026-02-25T08:46:01.366 INFO:tasks.rados.rados.0.trial169.stdout:953: finishing write tid 5 to trial16941936-21 2026-02-25T08:46:01.366 INFO:tasks.rados.rados.0.trial169.stdout:953: oid 21 updating version 0 to 88 2026-02-25T08:46:01.366 INFO:tasks.rados.rados.0.trial169.stdout:953: oid 21 updating version 88 to 91 2026-02-25T08:46:01.366 INFO:tasks.rados.rados.0.trial169.stdout:953: oid 21 version 91 is already newer than 91 2026-02-25T08:46:01.366 INFO:tasks.rados.rados.0.trial169.stdout:953: oid 21 version 91 is already newer than 91 2026-02-25T08:46:01.366 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 21 v 91 (ObjNum 453 snap 122 seq_num 453) dirty exists 2026-02-25T08:46:01.366 INFO:tasks.rados.rados.0.trial169.stdout:953: left oid 21 (ObjNum 453 snap 122 seq_num 453) 2026-02-25T08:46:01.366 INFO:tasks.rados.rados.0.trial169.stdout:954: finishing write tid 4 to trial16941936-14 2026-02-25T08:46:01.366 INFO:tasks.rados.rados.0.trial169.stdout:954: finishing write tid 5 to trial16941936-14 2026-02-25T08:46:01.367 INFO:tasks.rados.rados.0.trial169.stdout:954: oid 14 updating version 0 to 158 2026-02-25T08:46:01.367 INFO:tasks.rados.rados.0.trial169.stdout:954: oid 14 version 158 is already newer than 158 2026-02-25T08:46:01.367 INFO:tasks.rados.rados.0.trial169.stdout:954: oid 14 version 158 is already newer than 158 2026-02-25T08:46:01.367 INFO:tasks.rados.rados.0.trial169.stdout:954: oid 14 version 158 is already newer than 158 2026-02-25T08:46:01.367 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 14 v 158 (ObjNum 454 snap 122 seq_num 454) dirty exists 2026-02-25T08:46:01.367 INFO:tasks.rados.rados.0.trial169.stdout:954: left oid 14 (ObjNum 454 snap 122 seq_num 454) 2026-02-25T08:46:01.367 INFO:tasks.rados.rados.0.trial169.stdout:952: finishing write tid 1 to trial16941936-5 2026-02-25T08:46:01.367 INFO:tasks.rados.rados.0.trial169.stdout:952: finishing write tid 2 to trial16941936-5 2026-02-25T08:46:01.367 INFO:tasks.rados.rados.0.trial169.stdout:952: finishing write tid 3 to trial16941936-5 2026-02-25T08:46:01.367 INFO:tasks.rados.rados.0.trial169.stdout:952: finishing write tid 4 to trial16941936-5 2026-02-25T08:46:01.367 INFO:tasks.rados.rados.0.trial169.stdout:952: finishing write tid 5 to trial16941936-5 2026-02-25T08:46:01.367 INFO:tasks.rados.rados.0.trial169.stdout:952: oid 5 updating version 0 to 203 2026-02-25T08:46:01.367 INFO:tasks.rados.rados.0.trial169.stdout:952: oid 5 updating version 203 to 206 2026-02-25T08:46:01.368 INFO:tasks.rados.rados.0.trial169.stdout:952: oid 5 version 206 is already newer than 206 2026-02-25T08:46:01.368 INFO:tasks.rados.rados.0.trial169.stdout:952: oid 5 version 206 is already newer than 206 2026-02-25T08:46:01.368 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 5 v 206 (ObjNum 452 snap 122 seq_num 452) dirty exists 2026-02-25T08:46:01.368 INFO:tasks.rados.rados.0.trial169.stdout:952: left oid 5 (ObjNum 452 snap 122 seq_num 452) 2026-02-25T08:46:02.200 INFO:tasks.rados.rados.0.trial169.stdout:951: done (5 left) 2026-02-25T08:46:02.201 INFO:tasks.rados.rados.0.trial169.stdout:952: done (4 left) 2026-02-25T08:46:02.201 INFO:tasks.rados.rados.0.trial169.stdout:953: done (3 left) 2026-02-25T08:46:02.201 INFO:tasks.rados.rados.0.trial169.stdout:954: done (2 left) 2026-02-25T08:46:02.201 INFO:tasks.rados.rados.0.trial169.stdout:955: done (1 left) 2026-02-25T08:46:02.201 INFO:tasks.rados.rados.0.trial169.stdout:956: done (0 left) 2026-02-25T08:46:02.201 INFO:tasks.rados.rados.0.trial169.stdout:957: read oid 30 snap -1 2026-02-25T08:46:02.202 INFO:tasks.rados.rados.0.trial169.stdout:957: expect (ObjNum 339 snap 94 seq_num 339) 2026-02-25T08:46:02.202 INFO:tasks.rados.rados.0.trial169.stdout:958: delete oid 31 current snap is 123 2026-02-25T08:46:02.229 INFO:tasks.rados.rados.0.trial169.stdout:957: expect (ObjNum 339 snap 94 seq_num 339) 2026-02-25T08:46:02.231 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1276710 2026-02-25T08:46:02.298 INFO:tasks.rados.rados.0.trial169.stdout:957: done (1 left) 2026-02-25T08:46:02.299 INFO:tasks.rados.rados.0.trial169.stdout:958: done (0 left) 2026-02-25T08:46:02.299 INFO:tasks.rados.rados.0.trial169.stdout:959: delete oid 9 current snap is 123 2026-02-25T08:46:02.301 INFO:tasks.rados.rados.0.trial169.stdout:959: done (0 left) 2026-02-25T08:46:02.302 INFO:tasks.rados.rados.0.trial169.stdout:960: read oid 7 snap -1 2026-02-25T08:46:02.302 INFO:tasks.rados.rados.0.trial169.stdout:960: expect deleted 2026-02-25T08:46:02.302 INFO:tasks.rados.rados.0.trial169.stdout:961: write oid 25 current snap is 123 2026-02-25T08:46:02.302 INFO:tasks.rados.rados.0.trial169.stdout:961: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2026-02-25T08:46:02.306 INFO:tasks.rados.rados.0.trial169.stdout:961: writing trial16941936-25 from 497606 to 1142828 tid 1 2026-02-25T08:46:02.311 INFO:tasks.rados.rados.0.trial169.stdout:961: writing trial16941936-25 from 1934400 to 2563894 tid 2 2026-02-25T08:46:02.316 INFO:tasks.rados.rados.0.trial169.stdout:961: writing trial16941936-25 from 3243966 to 3697606 tid 3 2026-02-25T08:46:02.316 INFO:tasks.rados.rados.0.trial169.stdout:962: read oid 43 snap 122 2026-02-25T08:46:02.316 INFO:tasks.rados.rados.0.trial169.stdout:962: expect (ObjNum 436 snap 116 seq_num 436) 2026-02-25T08:46:02.316 INFO:tasks.rados.rados.0.trial169.stdout:963: delete oid 48 current snap is 123 2026-02-25T08:46:02.335 INFO:tasks.rados.rados.0.trial169.stdout:962: expect (ObjNum 436 snap 116 seq_num 436) 2026-02-25T08:46:02.335 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 8240 2026-02-25T08:46:02.390 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:02.485 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 3033245 2026-02-25T08:46:02.488 INFO:tasks.rados.rados.0.trial169.stdout:961: finishing write tid 1 to trial16941936-25 2026-02-25T08:46:02.488 INFO:tasks.rados.rados.0.trial169.stdout:961: finishing write tid 2 to trial16941936-25 2026-02-25T08:46:02.488 INFO:tasks.rados.rados.0.trial169.stdout:961: finishing write tid 3 to trial16941936-25 2026-02-25T08:46:02.488 INFO:tasks.rados.rados.0.trial169.stdout:961: finishing write tid 4 to trial16941936-25 2026-02-25T08:46:02.488 INFO:tasks.rados.rados.0.trial169.stdout:961: finishing write tid 5 to trial16941936-25 2026-02-25T08:46:02.488 INFO:tasks.rados.rados.0.trial169.stdout:961: oid 25 updating version 0 to 94 2026-02-25T08:46:02.489 INFO:tasks.rados.rados.0.trial169.stdout:961: oid 25 version 94 is already newer than 94 2026-02-25T08:46:02.489 INFO:tasks.rados.rados.0.trial169.stdout:961: oid 25 version 94 is already newer than 94 2026-02-25T08:46:02.489 INFO:tasks.rados.rados.0.trial169.stdout:961: oid 25 version 94 is already newer than 94 2026-02-25T08:46:02.489 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 25 v 94 (ObjNum 458 snap 123 seq_num 458) dirty exists 2026-02-25T08:46:02.489 INFO:tasks.rados.rados.0.trial169.stdout:961: left oid 25 (ObjNum 458 snap 123 seq_num 458) 2026-02-25T08:46:02.489 INFO:tasks.rados.rados.0.trial169.stdout:960: done (3 left) 2026-02-25T08:46:02.489 INFO:tasks.rados.rados.0.trial169.stdout:961: done (2 left) 2026-02-25T08:46:02.489 INFO:tasks.rados.rados.0.trial169.stdout:962: done (1 left) 2026-02-25T08:46:02.489 INFO:tasks.rados.rados.0.trial169.stdout:963: done (0 left) 2026-02-25T08:46:02.489 INFO:tasks.rados.rados.0.trial169.stdout:964: delete oid 49 current snap is 123 2026-02-25T08:46:02.498 INFO:tasks.rados.rados.0.trial169.stdout:964: done (0 left) 2026-02-25T08:46:02.498 INFO:tasks.rados.rados.0.trial169.stdout:965: read oid 48 snap -1 2026-02-25T08:46:02.498 INFO:tasks.rados.rados.0.trial169.stdout:965: expect deleted 2026-02-25T08:46:02.498 INFO:tasks.rados.rados.0.trial169.stdout:966: delete oid 29 current snap is 123 2026-02-25T08:46:02.507 INFO:tasks.rados.rados.0.trial169.stdout:965: done (1 left) 2026-02-25T08:46:02.507 INFO:tasks.rados.rados.0.trial169.stdout:966: done (0 left) 2026-02-25T08:46:02.507 INFO:tasks.rados.rados.0.trial169.stdout:967: delete oid 48 current snap is 123 2026-02-25T08:46:02.509 INFO:tasks.rados.rados.0.trial169.stdout:967: done (0 left) 2026-02-25T08:46:02.509 INFO:tasks.rados.rados.0.trial169.stdout:968: write oid 46 current snap is 123 2026-02-25T08:46:02.509 INFO:tasks.rados.rados.0.trial169.stdout:968: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2026-02-25T08:46:02.514 INFO:tasks.rados.rados.0.trial169.stdout:968: writing trial16941936-46 from 581641 to 1203108 tid 1 2026-02-25T08:46:02.520 INFO:tasks.rados.rados.0.trial169.stdout:968: writing trial16941936-46 from 1994104 to 2774594 tid 2 2026-02-25T08:46:02.524 INFO:tasks.rados.rados.0.trial169.stdout:968: writing trial16941936-46 from 3432022 to 3781641 tid 3 2026-02-25T08:46:02.524 INFO:tasks.rados.rados.0.trial169.stdout:969: write oid 25 current snap is 123 2026-02-25T08:46:02.524 INFO:tasks.rados.rados.0.trial169.stdout:969: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2026-02-25T08:46:02.528 INFO:tasks.rados.rados.0.trial169.stdout:969: writing trial16941936-25 from 598448 to 1011517 tid 1 2026-02-25T08:46:02.532 INFO:tasks.rados.rados.0.trial169.stdout:969: writing trial16941936-25 from 1768939 to 2293087 tid 2 2026-02-25T08:46:02.536 INFO:tasks.rados.rados.0.trial169.stdout:969: writing trial16941936-25 from 2775798 to 3357659 tid 3 2026-02-25T08:46:02.538 INFO:tasks.rados.rados.0.trial169.stdout:969: writing trial16941936-25 from 3781488 to 3798448 tid 4 2026-02-25T08:46:02.538 INFO:tasks.rados.rados.0.trial169.stdout:970: write (excl) oid 15 current snap is 123 2026-02-25T08:46:02.538 INFO:tasks.rados.rados.0.trial169.stdout:970: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2026-02-25T08:46:02.542 INFO:tasks.rados.rados.0.trial169.stdout:970: writing trial16941936-15 from 615255 to 1103573 tid 1 2026-02-25T08:46:02.548 INFO:tasks.rados.rados.0.trial169.stdout:970: writing trial16941936-15 from 1827421 to 2611580 tid 2 2026-02-25T08:46:02.553 INFO:tasks.rados.rados.0.trial169.stdout:970: writing trial16941936-15 from 3203221 to 3815255 tid 3 2026-02-25T08:46:02.553 INFO:tasks.rados.rados.0.trial169.stdout:971: snap_remove snap 121 2026-02-25T08:46:02.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~237s 2026-02-25T08:46:03.201 INFO:tasks.rados.rados.0.trial169.stdout:971: done (3 left) 2026-02-25T08:46:03.201 INFO:tasks.rados.rados.0.trial169.stdout:972: snap_remove snap 120 2026-02-25T08:46:03.322 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:03.398 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:03.395+0000 7f6e8a45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:03.400 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:03.399+0000 7f6e8a45a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:03.404 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:03.441 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:03.442 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:04.204 INFO:tasks.rados.rados.0.trial169.stdout:972: done (3 left) 2026-02-25T08:46:04.205 INFO:tasks.rados.rados.0.trial169.stdout:973: snap_remove snap 117 2026-02-25T08:46:05.211 INFO:tasks.rados.rados.0.trial169.stdout:973: done (3 left) 2026-02-25T08:46:05.211 INFO:tasks.rados.rados.0.trial169.stdout:974: delete oid 24 current snap is 123 2026-02-25T08:46:05.211 INFO:tasks.rados.rados.0.trial169.stdout:968: finishing write tid 1 to trial16941936-46 2026-02-25T08:46:05.211 INFO:tasks.rados.rados.0.trial169.stdout:968: finishing write tid 2 to trial16941936-46 2026-02-25T08:46:05.211 INFO:tasks.rados.rados.0.trial169.stdout:968: finishing write tid 3 to trial16941936-46 2026-02-25T08:46:05.211 INFO:tasks.rados.rados.0.trial169.stdout:969: finishing write tid 1 to trial16941936-25 2026-02-25T08:46:05.212 INFO:tasks.rados.rados.0.trial169.stdout:968: finishing write tid 4 to trial16941936-46 2026-02-25T08:46:05.212 INFO:tasks.rados.rados.0.trial169.stdout:968: finishing write tid 5 to trial16941936-46 2026-02-25T08:46:05.212 INFO:tasks.rados.rados.0.trial169.stdout:968: oid 46 updating version 0 to 162 2026-02-25T08:46:05.212 INFO:tasks.rados.rados.0.trial169.stdout:968: oid 46 version 162 is already newer than 162 2026-02-25T08:46:05.212 INFO:tasks.rados.rados.0.trial169.stdout:968: oid 46 version 162 is already newer than 162 2026-02-25T08:46:05.212 INFO:tasks.rados.rados.0.trial169.stdout:968: oid 46 version 162 is already newer than 162 2026-02-25T08:46:05.212 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 46 v 162 (ObjNum 463 snap 123 seq_num 463) dirty exists 2026-02-25T08:46:05.212 INFO:tasks.rados.rados.0.trial169.stdout:968: left oid 46 (ObjNum 463 snap 123 seq_num 463) 2026-02-25T08:46:05.212 INFO:tasks.rados.rados.0.trial169.stdout:969: finishing write tid 2 to trial16941936-25 2026-02-25T08:46:05.212 INFO:tasks.rados.rados.0.trial169.stdout:970: finishing write tid 1 to trial16941936-15 2026-02-25T08:46:05.212 INFO:tasks.rados.rados.0.trial169.stdout:969: finishing write tid 3 to trial16941936-25 2026-02-25T08:46:05.212 INFO:tasks.rados.rados.0.trial169.stdout:969: finishing write tid 4 to trial16941936-25 2026-02-25T08:46:05.212 INFO:tasks.rados.rados.0.trial169.stdout:969: finishing write tid 5 to trial16941936-25 2026-02-25T08:46:05.213 INFO:tasks.rados.rados.0.trial169.stdout:969: finishing write tid 6 to trial16941936-25 2026-02-25T08:46:05.213 INFO:tasks.rados.rados.0.trial169.stdout:969: oid 25 updating version 0 to 99 2026-02-25T08:46:05.213 INFO:tasks.rados.rados.0.trial169.stdout:969: oid 25 version 99 is already newer than 99 2026-02-25T08:46:05.213 INFO:tasks.rados.rados.0.trial169.stdout:969: oid 25 version 99 is already newer than 99 2026-02-25T08:46:05.213 INFO:tasks.rados.rados.0.trial169.stdout:969: oid 25 version 99 is already newer than 99 2026-02-25T08:46:05.213 INFO:tasks.rados.rados.0.trial169.stdout:969: oid 25 version 99 is already newer than 99 2026-02-25T08:46:05.213 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 25 v 99 (ObjNum 464 snap 123 seq_num 464) dirty exists 2026-02-25T08:46:05.213 INFO:tasks.rados.rados.0.trial169.stdout:969: left oid 25 (ObjNum 464 snap 123 seq_num 464) 2026-02-25T08:46:05.213 INFO:tasks.rados.rados.0.trial169.stdout:970: finishing write tid 2 to trial16941936-15 2026-02-25T08:46:05.213 INFO:tasks.rados.rados.0.trial169.stdout:970: finishing write tid 3 to trial16941936-15 2026-02-25T08:46:05.213 INFO:tasks.rados.rados.0.trial169.stdout:970: finishing write tid 4 to trial16941936-15 2026-02-25T08:46:05.213 INFO:tasks.rados.rados.0.trial169.stdout:970: finishing write tid 5 to trial16941936-15 2026-02-25T08:46:05.213 INFO:tasks.rados.rados.0.trial169.stdout:970: oid 15 updating version 0 to 166 2026-02-25T08:46:05.213 INFO:tasks.rados.rados.0.trial169.stdout:970: oid 15 version 166 is already newer than 166 2026-02-25T08:46:05.214 INFO:tasks.rados.rados.0.trial169.stdout:970: oid 15 version 166 is already newer than 166 2026-02-25T08:46:05.214 INFO:tasks.rados.rados.0.trial169.stdout:970: oid 15 version 166 is already newer than 163 2026-02-25T08:46:05.214 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 15 v 166 (ObjNum 465 snap 123 seq_num 465) dirty exists 2026-02-25T08:46:05.214 INFO:tasks.rados.rados.0.trial169.stdout:970: left oid 15 (ObjNum 465 snap 123 seq_num 465) 2026-02-25T08:46:05.278 INFO:tasks.rados.rados.0.trial169.stdout:968: done (3 left) 2026-02-25T08:46:05.279 INFO:tasks.rados.rados.0.trial169.stdout:969: done (2 left) 2026-02-25T08:46:05.279 INFO:tasks.rados.rados.0.trial169.stdout:970: done (1 left) 2026-02-25T08:46:05.279 INFO:tasks.rados.rados.0.trial169.stdout:974: done (0 left) 2026-02-25T08:46:05.279 INFO:tasks.rados.rados.0.trial169.stdout:975: read oid 29 snap 122 2026-02-25T08:46:05.279 INFO:tasks.rados.rados.0.trial169.stdout:975: expect deleted 2026-02-25T08:46:05.279 INFO:tasks.rados.rados.0.trial169.stdout:976: snap_create 2026-02-25T08:46:05.443 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:05.520 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:05.518+0000 7f8b127ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:05.523 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:05.521+0000 7f8b127ed640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:05.526 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:05.566 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:06.003 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:46:06.075 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:06.078 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:06.078 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:46:06.213 INFO:tasks.rados.rados.0.trial169.stdout:975: done (1 left) 2026-02-25T08:46:06.213 INFO:tasks.rados.rados.0.trial169.stdout:976: done (0 left) 2026-02-25T08:46:06.213 INFO:tasks.rados.rados.0.trial169.stdout:977: write (excl) oid 26 current snap is 124 2026-02-25T08:46:06.214 INFO:tasks.rados.rados.0.trial169.stdout:977: seq_num 467 ranges {648869=638816,1828032=787828,3141714=532030} 2026-02-25T08:46:06.218 INFO:tasks.rados.rados.0.trial169.stdout:977: writing trial16941936-26 from 648869 to 1287685 tid 1 2026-02-25T08:46:06.223 INFO:tasks.rados.rados.0.trial169.stdout:977: writing trial16941936-26 from 1828032 to 2615860 tid 2 2026-02-25T08:46:06.228 INFO:tasks.rados.rados.0.trial169.stdout:977: writing trial16941936-26 from 3141714 to 3673744 tid 3 2026-02-25T08:46:06.228 INFO:tasks.rados.rados.0.trial169.stdout:978: read oid 6 snap -1 2026-02-25T08:46:06.228 INFO:tasks.rados.rados.0.trial169.stdout:978: expect deleted 2026-02-25T08:46:06.228 INFO:tasks.rados.rados.0.trial169.stdout:979: delete oid 4 current snap is 124 2026-02-25T08:46:06.241 INFO:tasks.rados.rados.0.trial169.stdout:978: done (2 left) 2026-02-25T08:46:06.241 INFO:tasks.rados.rados.0.trial169.stdout:979: done (1 left) 2026-02-25T08:46:06.241 INFO:tasks.rados.rados.0.trial169.stdout:980: delete oid 49 current snap is 124 2026-02-25T08:46:06.258 INFO:tasks.rados.rados.0.trial169.stdout:977: finishing write tid 1 to trial16941936-26 2026-02-25T08:46:06.273 INFO:tasks.rados.rados.0.trial169.stdout:977: finishing write tid 2 to trial16941936-26 2026-02-25T08:46:06.279 INFO:tasks.rados.rados.0.trial169.stdout:977: finishing write tid 3 to trial16941936-26 2026-02-25T08:46:06.284 INFO:tasks.rados.rados.0.trial169.stdout:977: finishing write tid 4 to trial16941936-26 2026-02-25T08:46:06.286 INFO:tasks.rados.rados.0.trial169.stdout:977: finishing write tid 5 to trial16941936-26 2026-02-25T08:46:06.286 INFO:tasks.rados.rados.0.trial169.stdout:977: oid 26 updating version 0 to 426 2026-02-25T08:46:06.286 INFO:tasks.rados.rados.0.trial169.stdout:977: oid 26 updating version 426 to 429 2026-02-25T08:46:06.286 INFO:tasks.rados.rados.0.trial169.stdout:977: oid 26 version 429 is already newer than 429 2026-02-25T08:46:06.286 INFO:tasks.rados.rados.0.trial169.stdout:977: oid 26 version 429 is already newer than 429 2026-02-25T08:46:06.286 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 26 v 429 (ObjNum 467 snap 124 seq_num 467) dirty exists 2026-02-25T08:46:06.286 INFO:tasks.rados.rados.0.trial169.stdout:977: left oid 26 (ObjNum 467 snap 124 seq_num 467) 2026-02-25T08:46:06.287 INFO:tasks.rados.rados.0.trial169.stdout:977: done (1 left) 2026-02-25T08:46:06.287 INFO:tasks.rados.rados.0.trial169.stdout:980: done (0 left) 2026-02-25T08:46:06.287 INFO:tasks.rados.rados.0.trial169.stdout:981: write oid 30 current snap is 124 2026-02-25T08:46:06.287 INFO:tasks.rados.rados.0.trial169.stdout:981: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2026-02-25T08:46:06.292 INFO:tasks.rados.rados.0.trial169.stdout:981: writing trial16941936-30 from 699290 to 1163853 tid 1 2026-02-25T08:46:06.296 INFO:tasks.rados.rados.0.trial169.stdout:981: writing trial16941936-30 from 1603478 to 2138633 tid 2 2026-02-25T08:46:06.301 INFO:tasks.rados.rados.0.trial169.stdout:981: writing trial16941936-30 from 2823983 to 3589829 tid 3 2026-02-25T08:46:06.301 INFO:tasks.rados.rados.0.trial169.stdout:982: snap_create 2026-02-25T08:46:06.323 INFO:tasks.rados.rados.0.trial169.stdout:981: finishing write tid 1 to trial16941936-30 2026-02-25T08:46:06.330 INFO:tasks.rados.rados.0.trial169.stdout:981: finishing write tid 2 to trial16941936-30 2026-02-25T08:46:06.335 INFO:tasks.rados.rados.0.trial169.stdout:981: finishing write tid 3 to trial16941936-30 2026-02-25T08:46:06.343 INFO:tasks.rados.rados.0.trial169.stdout:981: finishing write tid 4 to trial16941936-30 2026-02-25T08:46:06.345 INFO:tasks.rados.rados.0.trial169.stdout:981: finishing write tid 5 to trial16941936-30 2026-02-25T08:46:06.345 INFO:tasks.rados.rados.0.trial169.stdout:981: oid 30 updating version 0 to 147 2026-02-25T08:46:06.345 INFO:tasks.rados.rados.0.trial169.stdout:981: oid 30 version 147 is already newer than 147 2026-02-25T08:46:06.345 INFO:tasks.rados.rados.0.trial169.stdout:981: oid 30 version 147 is already newer than 147 2026-02-25T08:46:06.345 INFO:tasks.rados.rados.0.trial169.stdout:981: oid 30 version 147 is already newer than 147 2026-02-25T08:46:06.345 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 30 v 147 (ObjNum 470 snap 124 seq_num 470) dirty exists 2026-02-25T08:46:06.346 INFO:tasks.rados.rados.0.trial169.stdout:981: left oid 30 (ObjNum 470 snap 124 seq_num 470) 2026-02-25T08:46:07.216 INFO:tasks.rados.rados.0.trial169.stdout:981: done (1 left) 2026-02-25T08:46:07.217 INFO:tasks.rados.rados.0.trial169.stdout:982: done (0 left) 2026-02-25T08:46:07.217 INFO:tasks.rados.rados.0.trial169.stdout:983: delete oid 16 current snap is 125 2026-02-25T08:46:07.232 INFO:tasks.rados.rados.0.trial169.stdout:983: done (0 left) 2026-02-25T08:46:07.232 INFO:tasks.rados.rados.0.trial169.stdout:984: snap_create 2026-02-25T08:46:07.567 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:07.646 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:07.643+0000 7f6c2cb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:07.648 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:07.646+0000 7f6c2cb0b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:07.652 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:07.691 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:08.160 INFO:tasks.rados.rados.0.trial169.stdout:984: done (0 left) 2026-02-25T08:46:08.160 INFO:tasks.rados.rados.0.trial169.stdout:985: write (excl) oid 50 current snap is 126 2026-02-25T08:46:08.160 INFO:tasks.rados.rados.0.trial169.stdout:985: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2026-02-25T08:46:08.165 INFO:tasks.rados.rados.0.trial169.stdout:985: writing trial16941936-50 from 732904 to 1464318 tid 1 2026-02-25T08:46:08.170 INFO:tasks.rados.rados.0.trial169.stdout:985: writing trial16941936-50 from 2120442 to 2659266 tid 2 2026-02-25T08:46:08.175 INFO:tasks.rados.rados.0.trial169.stdout:985: writing trial16941936-50 from 3278829 to 3783572 tid 3 2026-02-25T08:46:08.175 INFO:tasks.rados.rados.0.trial169.stdout:986: read oid 37 snap 119 2026-02-25T08:46:08.175 INFO:tasks.rados.rados.0.trial169.stdout:986: expect deleted 2026-02-25T08:46:08.175 INFO:tasks.rados.rados.0.trial169.stdout:987: delete oid 41 current snap is 126 2026-02-25T08:46:08.184 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~242s 2026-02-25T08:46:08.203 INFO:tasks.rados.rados.0.trial169.stdout:985: finishing write tid 1 to trial16941936-50 2026-02-25T08:46:08.204 INFO:tasks.rados.rados.0.trial169.stdout:986: done (2 left) 2026-02-25T08:46:08.204 INFO:tasks.rados.rados.0.trial169.stdout:987: done (1 left) 2026-02-25T08:46:08.204 INFO:tasks.rados.rados.0.trial169.stdout:988: read oid 42 snap 119 2026-02-25T08:46:08.204 INFO:tasks.rados.rados.0.trial169.stdout:988: expect (ObjNum 324 snap 87 seq_num 324) 2026-02-25T08:46:08.204 INFO:tasks.rados.rados.0.trial169.stdout:989: write (excl) oid 27 current snap is 126 2026-02-25T08:46:08.204 INFO:tasks.rados.rados.0.trial169.stdout:989: seq_num 474 ranges {766518=481912,1837406=542493,2933675=527287,3876584=89934} 2026-02-25T08:46:08.209 INFO:tasks.rados.rados.0.trial169.stdout:989: writing trial16941936-27 from 766518 to 1248430 tid 1 2026-02-25T08:46:08.214 INFO:tasks.rados.rados.0.trial169.stdout:989: writing trial16941936-27 from 1837406 to 2379899 tid 2 2026-02-25T08:46:08.218 INFO:tasks.rados.rados.0.trial169.stdout:989: writing trial16941936-27 from 2933675 to 3460962 tid 3 2026-02-25T08:46:08.220 INFO:tasks.rados.rados.0.trial169.stdout:989: writing trial16941936-27 from 3876584 to 3966518 tid 4 2026-02-25T08:46:08.220 INFO:tasks.rados.rados.0.trial169.stdout:990: read oid 30 snap 123 2026-02-25T08:46:08.221 INFO:tasks.rados.rados.0.trial169.stdout:990: expect (ObjNum 339 snap 94 seq_num 339) 2026-02-25T08:46:08.221 INFO:tasks.rados.rados.0.trial169.stdout:991: read oid 3 snap -1 2026-02-25T08:46:08.221 INFO:tasks.rados.rados.0.trial169.stdout:991: expect (ObjNum 428 snap 115 seq_num 428) 2026-02-25T08:46:08.221 INFO:tasks.rados.rados.0.trial169.stdout:992: write oid 28 current snap is 126 2026-02-25T08:46:08.221 INFO:tasks.rados.rados.0.trial169.stdout:992: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2026-02-25T08:46:08.225 INFO:tasks.rados.rados.0.trial169.stdout:992: writing trial16941936-28 from 783325 to 1340486 tid 1 2026-02-25T08:46:08.229 INFO:tasks.rados.rados.0.trial169.stdout:992: writing trial16941936-28 from 1779535 to 2465686 tid 2 2026-02-25T08:46:08.234 INFO:tasks.rados.rados.0.trial169.stdout:992: writing trial16941936-28 from 3244745 to 3699657 tid 3 2026-02-25T08:46:08.234 INFO:tasks.rados.rados.0.trial169.stdout:993: snap_remove snap 118 2026-02-25T08:46:09.223 INFO:tasks.rados.rados.0.trial169.stdout:993: done (6 left) 2026-02-25T08:46:09.223 INFO:tasks.rados.rados.0.trial169.stdout:994: delete oid 35 current snap is 126 2026-02-25T08:46:09.223 INFO:tasks.rados.rados.0.trial169.stdout:985: finishing write tid 2 to trial16941936-50 2026-02-25T08:46:09.223 INFO:tasks.rados.rados.0.trial169.stdout:985: finishing write tid 3 to trial16941936-50 2026-02-25T08:46:09.223 INFO:tasks.rados.rados.0.trial169.stdout:988: expect (ObjNum 324 snap 87 seq_num 324) 2026-02-25T08:46:09.242 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 3066238 2026-02-25T08:46:09.291 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:09.332 INFO:tasks.rados.rados.0.trial169.stdout:985: finishing write tid 4 to trial16941936-50 2026-02-25T08:46:09.332 INFO:tasks.rados.rados.0.trial169.stdout:985: finishing write tid 5 to trial16941936-50 2026-02-25T08:46:09.332 INFO:tasks.rados.rados.0.trial169.stdout:985: oid 50 updating version 0 to 529 2026-02-25T08:46:09.332 INFO:tasks.rados.rados.0.trial169.stdout:985: oid 50 version 529 is already newer than 529 2026-02-25T08:46:09.332 INFO:tasks.rados.rados.0.trial169.stdout:985: oid 50 version 529 is already newer than 529 2026-02-25T08:46:09.332 INFO:tasks.rados.rados.0.trial169.stdout:985: oid 50 version 529 is already newer than 526 2026-02-25T08:46:09.332 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 50 v 529 (ObjNum 472 snap 126 seq_num 472) dirty exists 2026-02-25T08:46:09.332 INFO:tasks.rados.rados.0.trial169.stdout:985: left oid 50 (ObjNum 472 snap 126 seq_num 472) 2026-02-25T08:46:09.332 INFO:tasks.rados.rados.0.trial169.stdout:992: finishing write tid 1 to trial16941936-28 2026-02-25T08:46:09.333 INFO:tasks.rados.rados.0.trial169.stdout:985: done (6 left) 2026-02-25T08:46:09.333 INFO:tasks.rados.rados.0.trial169.stdout:988: done (5 left) 2026-02-25T08:46:09.333 INFO:tasks.rados.rados.0.trial169.stdout:994: done (4 left) 2026-02-25T08:46:09.333 INFO:tasks.rados.rados.0.trial169.stdout:995: delete oid 34 current snap is 126 2026-02-25T08:46:09.333 INFO:tasks.rados.rados.0.trial169.stdout:990: expect (ObjNum 339 snap 94 seq_num 339) 2026-02-25T08:46:09.362 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:09.419 INFO:tasks.rados.rados.0.trial169.stdout:992: finishing write tid 2 to trial16941936-28 2026-02-25T08:46:09.419 INFO:tasks.rados.rados.0.trial169.stdout:992: finishing write tid 3 to trial16941936-28 2026-02-25T08:46:09.419 INFO:tasks.rados.rados.0.trial169.stdout:990: done (4 left) 2026-02-25T08:46:09.420 INFO:tasks.rados.rados.0.trial169.stdout:995: done (3 left) 2026-02-25T08:46:09.420 INFO:tasks.rados.rados.0.trial169.stdout:996: write (excl) oid 32 current snap is 126 2026-02-25T08:46:09.420 INFO:tasks.rados.rados.0.trial169.stdout:989: finishing write tid 1 to trial16941936-27 2026-02-25T08:46:09.420 INFO:tasks.rados.rados.0.trial169.stdout:992: finishing write tid 4 to trial16941936-28 2026-02-25T08:46:09.420 INFO:tasks.rados.rados.0.trial169.stdout:992: finishing write tid 5 to trial16941936-28 2026-02-25T08:46:09.420 INFO:tasks.rados.rados.0.trial169.stdout:992: oid 28 updating version 0 to 58 2026-02-25T08:46:09.420 INFO:tasks.rados.rados.0.trial169.stdout:992: oid 28 version 58 is already newer than 58 2026-02-25T08:46:09.420 INFO:tasks.rados.rados.0.trial169.stdout:992: oid 28 version 58 is already newer than 58 2026-02-25T08:46:09.420 INFO:tasks.rados.rados.0.trial169.stdout:992: oid 28 version 58 is already newer than 58 2026-02-25T08:46:09.420 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 28 v 58 (ObjNum 475 snap 126 seq_num 475) dirty exists 2026-02-25T08:46:09.420 INFO:tasks.rados.rados.0.trial169.stdout:992: left oid 28 (ObjNum 475 snap 126 seq_num 475) 2026-02-25T08:46:09.421 INFO:tasks.rados.rados.0.trial169.stdout:991: expect (ObjNum 428 snap 115 seq_num 428) 2026-02-25T08:46:09.421 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:09.468 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1202858 2026-02-25T08:46:09.533 INFO:tasks.rados.rados.0.trial169.stdout:989: finishing write tid 2 to trial16941936-27 2026-02-25T08:46:09.533 INFO:tasks.rados.rados.0.trial169.stdout:989: finishing write tid 3 to trial16941936-27 2026-02-25T08:46:09.534 INFO:tasks.rados.rados.0.trial169.stdout:996: seq_num 478 ranges {433746=782908,1954981=78765} 2026-02-25T08:46:09.538 INFO:tasks.rados.rados.0.trial169.stdout:996: writing trial16941936-32 from 433746 to 1216654 tid 1 2026-02-25T08:46:09.542 INFO:tasks.rados.rados.0.trial169.stdout:996: writing trial16941936-32 from 1954981 to 2033746 tid 2 2026-02-25T08:46:09.542 INFO:tasks.rados.rados.0.trial169.stdout:991: done (3 left) 2026-02-25T08:46:09.542 INFO:tasks.rados.rados.0.trial169.stdout:992: done (2 left) 2026-02-25T08:46:09.542 INFO:tasks.rados.rados.0.trial169.stdout:997: snap_create 2026-02-25T08:46:09.542 INFO:tasks.rados.rados.0.trial169.stdout:989: finishing write tid 4 to trial16941936-27 2026-02-25T08:46:09.542 INFO:tasks.rados.rados.0.trial169.stdout:989: finishing write tid 5 to trial16941936-27 2026-02-25T08:46:09.542 INFO:tasks.rados.rados.0.trial169.stdout:989: finishing write tid 6 to trial16941936-27 2026-02-25T08:46:09.542 INFO:tasks.rados.rados.0.trial169.stdout:989: oid 27 updating version 0 to 97 2026-02-25T08:46:09.542 INFO:tasks.rados.rados.0.trial169.stdout:989: oid 27 version 97 is already newer than 97 2026-02-25T08:46:09.543 INFO:tasks.rados.rados.0.trial169.stdout:989: oid 27 version 97 is already newer than 93 2026-02-25T08:46:09.543 INFO:tasks.rados.rados.0.trial169.stdout:989: oid 27 version 97 is already newer than 97 2026-02-25T08:46:09.543 INFO:tasks.rados.rados.0.trial169.stdout:989: oid 27 version 97 is already newer than 97 2026-02-25T08:46:09.543 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 27 v 97 (ObjNum 474 snap 126 seq_num 474) dirty exists 2026-02-25T08:46:09.543 INFO:tasks.rados.rados.0.trial169.stdout:989: left oid 27 (ObjNum 474 snap 126 seq_num 474) 2026-02-25T08:46:09.561 INFO:tasks.rados.rados.0.trial169.stdout:996: finishing write tid 1 to trial16941936-32 2026-02-25T08:46:09.571 INFO:tasks.rados.rados.0.trial169.stdout:996: finishing write tid 2 to trial16941936-32 2026-02-25T08:46:09.574 INFO:tasks.rados.rados.0.trial169.stdout:996: finishing write tid 3 to trial16941936-32 2026-02-25T08:46:09.575 INFO:tasks.rados.rados.0.trial169.stdout:996: finishing write tid 4 to trial16941936-32 2026-02-25T08:46:09.576 INFO:tasks.rados.rados.0.trial169.stdout:996: oid 32 updating version 0 to 94 2026-02-25T08:46:09.576 INFO:tasks.rados.rados.0.trial169.stdout:996: oid 32 version 94 is already newer than 94 2026-02-25T08:46:09.576 INFO:tasks.rados.rados.0.trial169.stdout:996: oid 32 version 94 is already newer than 92 2026-02-25T08:46:09.576 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 32 v 94 (ObjNum 478 snap 126 seq_num 478) dirty exists 2026-02-25T08:46:09.576 INFO:tasks.rados.rados.0.trial169.stdout:996: left oid 32 (ObjNum 478 snap 126 seq_num 478) 2026-02-25T08:46:09.693 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:09.770 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:09.768+0000 7fb297593640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:09.773 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:09.771+0000 7fb297593640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:09.776 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:09.814 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:09.815 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:10.225 INFO:tasks.rados.rados.0.trial169.stdout:989: done (2 left) 2026-02-25T08:46:10.226 INFO:tasks.rados.rados.0.trial169.stdout:996: done (1 left) 2026-02-25T08:46:10.226 INFO:tasks.rados.rados.0.trial169.stdout:997: done (0 left) 2026-02-25T08:46:10.226 INFO:tasks.rados.rados.0.trial169.stdout:998: read oid 17 snap -1 2026-02-25T08:46:10.226 INFO:tasks.rados.rados.0.trial169.stdout:998: expect (ObjNum 432 snap 115 seq_num 432) 2026-02-25T08:46:10.226 INFO:tasks.rados.rados.0.trial169.stdout:999: read oid 9 snap -1 2026-02-25T08:46:10.226 INFO:tasks.rados.rados.0.trial169.stdout:999: expect deleted 2026-02-25T08:46:10.226 INFO:tasks.rados.rados.0.trial169.stdout:1000: write (excl) oid 18 current snap is 127 2026-02-25T08:46:10.226 INFO:tasks.rados.rados.0.trial169.stdout:1000: seq_num 479 ranges {450553=574510,1613463=437090} 2026-02-25T08:46:10.229 INFO:tasks.rados.rados.0.trial169.stdout:1000: writing trial16941936-18 from 450553 to 1025063 tid 1 2026-02-25T08:46:10.233 INFO:tasks.rados.rados.0.trial169.stdout:1000: writing trial16941936-18 from 1613463 to 2050553 tid 2 2026-02-25T08:46:10.233 INFO:tasks.rados.rados.0.trial169.stdout:1001: snap_create 2026-02-25T08:46:10.274 INFO:tasks.rados.rados.0.trial169.stdout:998: expect (ObjNum 432 snap 115 seq_num 432) 2026-02-25T08:46:10.275 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:10.313 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:10.350 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:10.387 INFO:tasks.rados.rados.0.trial169.stdout:1000: finishing write tid 1 to trial16941936-18 2026-02-25T08:46:10.387 INFO:tasks.rados.rados.0.trial169.stdout:1000: finishing write tid 2 to trial16941936-18 2026-02-25T08:46:10.387 INFO:tasks.rados.rados.0.trial169.stdout:1000: finishing write tid 3 to trial16941936-18 2026-02-25T08:46:10.387 INFO:tasks.rados.rados.0.trial169.stdout:1000: finishing write tid 4 to trial16941936-18 2026-02-25T08:46:10.387 INFO:tasks.rados.rados.0.trial169.stdout:1000: oid 18 updating version 0 to 216 2026-02-25T08:46:10.387 INFO:tasks.rados.rados.0.trial169.stdout:1000: oid 18 version 216 is already newer than 214 2026-02-25T08:46:10.387 INFO:tasks.rados.rados.0.trial169.stdout:1000: oid 18 version 216 is already newer than 216 2026-02-25T08:46:10.387 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 18 v 216 (ObjNum 479 snap 127 seq_num 479) dirty exists 2026-02-25T08:46:10.387 INFO:tasks.rados.rados.0.trial169.stdout:1000: left oid 18 (ObjNum 479 snap 127 seq_num 479) 2026-02-25T08:46:11.079 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:46:11.151 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:11.155 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:46:11.227 INFO:tasks.rados.rados.0.trial169.stdout:998: done (3 left) 2026-02-25T08:46:11.227 INFO:tasks.rados.rados.0.trial169.stdout:999: done (2 left) 2026-02-25T08:46:11.228 INFO:tasks.rados.rados.0.trial169.stdout:1000: done (1 left) 2026-02-25T08:46:11.228 INFO:tasks.rados.rados.0.trial169.stdout:1001: done (0 left) 2026-02-25T08:46:11.228 INFO:tasks.rados.rados.0.trial169.stdout:1002: write oid 39 current snap is 128 2026-02-25T08:46:11.228 INFO:tasks.rados.rados.0.trial169.stdout:1002: seq_num 480 ranges {467360=649759,1671945=395415} 2026-02-25T08:46:11.231 INFO:tasks.rados.rados.0.trial169.stdout:1002: writing trial16941936-39 from 467360 to 1117119 tid 1 2026-02-25T08:46:11.235 INFO:tasks.rados.rados.0.trial169.stdout:1002: writing trial16941936-39 from 1671945 to 2067360 tid 2 2026-02-25T08:46:11.235 INFO:tasks.rados.rados.0.trial169.stdout:1003: snap_remove snap 125 2026-02-25T08:46:11.816 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:11.895 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:11.891+0000 7fef77ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:11.897 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:11.895+0000 7fef77ecb640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:11.900 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:11.940 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:11.941 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:12.233 INFO:tasks.rados.rados.0.trial169.stdout:1003: done (1 left) 2026-02-25T08:46:12.233 INFO:tasks.rados.rados.0.trial169.stdout:1004: delete oid 13 current snap is 128 2026-02-25T08:46:12.234 INFO:tasks.rados.rados.0.trial169.stdout:1002: finishing write tid 1 to trial16941936-39 2026-02-25T08:46:12.234 INFO:tasks.rados.rados.0.trial169.stdout:1002: finishing write tid 2 to trial16941936-39 2026-02-25T08:46:12.234 INFO:tasks.rados.rados.0.trial169.stdout:1002: finishing write tid 3 to trial16941936-39 2026-02-25T08:46:12.234 INFO:tasks.rados.rados.0.trial169.stdout:1002: finishing write tid 4 to trial16941936-39 2026-02-25T08:46:12.234 INFO:tasks.rados.rados.0.trial169.stdout:1002: oid 39 updating version 0 to 150 2026-02-25T08:46:12.234 INFO:tasks.rados.rados.0.trial169.stdout:1002: oid 39 version 150 is already newer than 150 2026-02-25T08:46:12.234 INFO:tasks.rados.rados.0.trial169.stdout:1002: oid 39 version 150 is already newer than 150 2026-02-25T08:46:12.234 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 39 v 150 (ObjNum 480 snap 128 seq_num 480) dirty exists 2026-02-25T08:46:12.234 INFO:tasks.rados.rados.0.trial169.stdout:1002: left oid 39 (ObjNum 480 snap 128 seq_num 480) 2026-02-25T08:46:12.260 INFO:tasks.rados.rados.0.trial169.stdout:1002: done (1 left) 2026-02-25T08:46:12.261 INFO:tasks.rados.rados.0.trial169.stdout:1004: done (0 left) 2026-02-25T08:46:12.261 INFO:tasks.rados.rados.0.trial169.stdout:1005: snap_remove snap 126 2026-02-25T08:46:13.236 INFO:tasks.rados.rados.0.trial169.stdout:1005: done (0 left) 2026-02-25T08:46:13.236 INFO:tasks.rados.rados.0.trial169.stdout:1006: snap_create 2026-02-25T08:46:13.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~248s 2026-02-25T08:46:13.942 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:14.020 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:14.017+0000 7f3a9f811640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:14.023 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:14.021+0000 7f3a9e588640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:14.026 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:14.067 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:14.239 INFO:tasks.rados.rados.0.trial169.stdout:1006: done (0 left) 2026-02-25T08:46:14.240 INFO:tasks.rados.rados.0.trial169.stdout:1007: read oid 43 snap -1 2026-02-25T08:46:14.240 INFO:tasks.rados.rados.0.trial169.stdout:1007: expect (ObjNum 436 snap 116 seq_num 436) 2026-02-25T08:46:14.240 INFO:tasks.rados.rados.0.trial169.stdout:1008: snap_remove snap 127 2026-02-25T08:46:15.243 INFO:tasks.rados.rados.0.trial169.stdout:1008: done (1 left) 2026-02-25T08:46:15.244 INFO:tasks.rados.rados.0.trial169.stdout:1009: read oid 45 snap -1 2026-02-25T08:46:15.244 INFO:tasks.rados.rados.0.trial169.stdout:1009: expect (ObjNum 401 snap 109 seq_num 401) 2026-02-25T08:46:15.244 INFO:tasks.rados.rados.0.trial169.stdout:1010: read oid 7 snap -1 2026-02-25T08:46:15.244 INFO:tasks.rados.rados.0.trial169.stdout:1010: expect deleted 2026-02-25T08:46:15.244 INFO:tasks.rados.rados.0.trial169.stdout:1011: write oid 41 current snap is 129 2026-02-25T08:46:15.244 INFO:tasks.rados.rados.0.trial169.stdout:1011: seq_num 482 ranges {500974=400257,1388909=557169} 2026-02-25T08:46:15.246 INFO:tasks.rados.rados.0.trial169.stdout:1011: writing trial16941936-41 from 500974 to 901231 tid 1 2026-02-25T08:46:15.251 INFO:tasks.rados.rados.0.trial169.stdout:1011: writing trial16941936-41 from 1388909 to 1946078 tid 2 2026-02-25T08:46:15.251 INFO:tasks.rados.rados.0.trial169.stdout:1012: write (excl) oid 33 current snap is 129 2026-02-25T08:46:15.251 INFO:tasks.rados.rados.0.trial169.stdout:1007: expect (ObjNum 436 snap 116 seq_num 436) 2026-02-25T08:46:15.251 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:15.307 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:15.363 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:15.418 INFO:tasks.rados.rados.0.trial169.stdout:1011: finishing write tid 1 to trial16941936-41 2026-02-25T08:46:15.419 INFO:tasks.rados.rados.0.trial169.stdout:1009: expect (ObjNum 401 snap 109 seq_num 401) 2026-02-25T08:46:15.440 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2429056 2026-02-25T08:46:15.454 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:15.546 INFO:tasks.rados.rados.0.trial169.stdout:1011: finishing write tid 2 to trial16941936-41 2026-02-25T08:46:15.546 INFO:tasks.rados.rados.0.trial169.stdout:1011: finishing write tid 3 to trial16941936-41 2026-02-25T08:46:15.546 INFO:tasks.rados.rados.0.trial169.stdout:1011: finishing write tid 4 to trial16941936-41 2026-02-25T08:46:15.546 INFO:tasks.rados.rados.0.trial169.stdout:1011: oid 41 updating version 0 to 444 2026-02-25T08:46:15.546 INFO:tasks.rados.rados.0.trial169.stdout:1011: oid 41 version 444 is already newer than 444 2026-02-25T08:46:15.546 INFO:tasks.rados.rados.0.trial169.stdout:1011: oid 41 version 444 is already newer than 444 2026-02-25T08:46:15.547 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 41 v 444 (ObjNum 482 snap 129 seq_num 482) dirty exists 2026-02-25T08:46:15.547 INFO:tasks.rados.rados.0.trial169.stdout:1011: left oid 41 (ObjNum 482 snap 129 seq_num 482) 2026-02-25T08:46:15.547 INFO:tasks.rados.rados.0.trial169.stdout:1012: seq_num 483 ranges {517781=475506,1731038=386743} 2026-02-25T08:46:15.550 INFO:tasks.rados.rados.0.trial169.stdout:1012: writing trial16941936-33 from 517781 to 993287 tid 1 2026-02-25T08:46:15.555 INFO:tasks.rados.rados.0.trial169.stdout:1012: writing trial16941936-33 from 1731038 to 2117781 tid 2 2026-02-25T08:46:15.555 INFO:tasks.rados.rados.0.trial169.stdout:1007: done (4 left) 2026-02-25T08:46:15.555 INFO:tasks.rados.rados.0.trial169.stdout:1009: done (3 left) 2026-02-25T08:46:15.555 INFO:tasks.rados.rados.0.trial169.stdout:1010: done (2 left) 2026-02-25T08:46:15.555 INFO:tasks.rados.rados.0.trial169.stdout:1011: done (1 left) 2026-02-25T08:46:15.555 INFO:tasks.rados.rados.0.trial169.stdout:1013: write (excl) oid 31 current snap is 129 2026-02-25T08:46:15.555 INFO:tasks.rados.rados.0.trial169.stdout:1013: seq_num 484 ranges {534588=550755,1789520=345068} 2026-02-25T08:46:15.558 INFO:tasks.rados.rados.0.trial169.stdout:1013: writing trial16941936-31 from 534588 to 1085343 tid 1 2026-02-25T08:46:15.563 INFO:tasks.rados.rados.0.trial169.stdout:1013: writing trial16941936-31 from 1789520 to 2134588 tid 2 2026-02-25T08:46:15.563 INFO:tasks.rados.rados.0.trial169.stdout:1014: read oid 19 snap 122 2026-02-25T08:46:15.563 INFO:tasks.rados.rados.0.trial169.stdout:1014: expect (ObjNum 410 snap 110 seq_num 410) 2026-02-25T08:46:15.563 INFO:tasks.rados.rados.0.trial169.stdout:1015: write oid 37 current snap is 129 2026-02-25T08:46:15.563 INFO:tasks.rados.rados.0.trial169.stdout:1015: seq_num 485 ranges {551395=626004,1848002=303393} 2026-02-25T08:46:15.567 INFO:tasks.rados.rados.0.trial169.stdout:1015: writing trial16941936-37 from 551395 to 1177399 tid 1 2026-02-25T08:46:15.571 INFO:tasks.rados.rados.0.trial169.stdout:1015: writing trial16941936-37 from 1848002 to 2151395 tid 2 2026-02-25T08:46:15.571 INFO:tasks.rados.rados.0.trial169.stdout:1016: read oid 45 snap -1 2026-02-25T08:46:15.571 INFO:tasks.rados.rados.0.trial169.stdout:1016: expect (ObjNum 401 snap 109 seq_num 401) 2026-02-25T08:46:15.571 INFO:tasks.rados.rados.0.trial169.stdout:1017: write oid 21 current snap is 129 2026-02-25T08:46:15.571 INFO:tasks.rados.rados.0.trial169.stdout:1017: seq_num 486 ranges {568202=701253,1906484=261718} 2026-02-25T08:46:15.575 INFO:tasks.rados.rados.0.trial169.stdout:1017: writing trial16941936-21 from 568202 to 1269455 tid 1 2026-02-25T08:46:15.579 INFO:tasks.rados.rados.0.trial169.stdout:1017: writing trial16941936-21 from 1906484 to 2168202 tid 2 2026-02-25T08:46:15.579 INFO:tasks.rados.rados.0.trial169.stdout:1018: read oid 30 snap -1 2026-02-25T08:46:15.579 INFO:tasks.rados.rados.0.trial169.stdout:1018: expect (ObjNum 470 snap 124 seq_num 470) 2026-02-25T08:46:15.579 INFO:tasks.rados.rados.0.trial169.stdout:1019: snap_create 2026-02-25T08:46:15.580 INFO:tasks.rados.rados.0.trial169.stdout:1013: finishing write tid 1 to trial16941936-31 2026-02-25T08:46:15.580 INFO:tasks.rados.rados.0.trial169.stdout:1014: expect (ObjNum 410 snap 110 seq_num 410) 2026-02-25T08:46:15.593 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1338915 2026-02-25T08:46:15.639 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1910052 2026-02-25T08:46:15.674 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2039251 2026-02-25T08:46:15.687 INFO:tasks.rados.rados.0.trial169.stdout:1015: finishing write tid 1 to trial16941936-37 2026-02-25T08:46:15.687 INFO:tasks.rados.rados.0.trial169.stdout:1013: finishing write tid 2 to trial16941936-31 2026-02-25T08:46:15.687 INFO:tasks.rados.rados.0.trial169.stdout:1018: expect (ObjNum 470 snap 124 seq_num 470) 2026-02-25T08:46:15.687 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:15.757 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 3633064 2026-02-25T08:46:15.781 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2854326 2026-02-25T08:46:15.795 INFO:tasks.rados.rados.0.trial169.stdout:1013: finishing write tid 3 to trial16941936-31 2026-02-25T08:46:15.795 INFO:tasks.rados.rados.0.trial169.stdout:1013: finishing write tid 4 to trial16941936-31 2026-02-25T08:46:15.796 INFO:tasks.rados.rados.0.trial169.stdout:1013: oid 31 updating version 0 to 447 2026-02-25T08:46:15.796 INFO:tasks.rados.rados.0.trial169.stdout:1013: oid 31 version 447 is already newer than 447 2026-02-25T08:46:15.796 INFO:tasks.rados.rados.0.trial169.stdout:1013: oid 31 version 447 is already newer than 445 2026-02-25T08:46:15.796 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 31 v 447 (ObjNum 484 snap 129 seq_num 484) dirty exists 2026-02-25T08:46:15.796 INFO:tasks.rados.rados.0.trial169.stdout:1013: left oid 31 (ObjNum 484 snap 129 seq_num 484) 2026-02-25T08:46:15.796 INFO:tasks.rados.rados.0.trial169.stdout:1015: finishing write tid 2 to trial16941936-37 2026-02-25T08:46:15.796 INFO:tasks.rados.rados.0.trial169.stdout:1017: finishing write tid 1 to trial16941936-21 2026-02-25T08:46:15.796 INFO:tasks.rados.rados.0.trial169.stdout:1015: finishing write tid 3 to trial16941936-37 2026-02-25T08:46:15.796 INFO:tasks.rados.rados.0.trial169.stdout:1017: finishing write tid 2 to trial16941936-21 2026-02-25T08:46:15.797 INFO:tasks.rados.rados.0.trial169.stdout:1015: finishing write tid 4 to trial16941936-37 2026-02-25T08:46:15.797 INFO:tasks.rados.rados.0.trial169.stdout:1015: oid 37 updating version 0 to 165 2026-02-25T08:46:15.797 INFO:tasks.rados.rados.0.trial169.stdout:1015: oid 37 version 165 is already newer than 165 2026-02-25T08:46:15.797 INFO:tasks.rados.rados.0.trial169.stdout:1015: oid 37 version 165 is already newer than 165 2026-02-25T08:46:15.797 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 37 v 165 (ObjNum 485 snap 129 seq_num 485) dirty exists 2026-02-25T08:46:15.797 INFO:tasks.rados.rados.0.trial169.stdout:1015: left oid 37 (ObjNum 485 snap 129 seq_num 485) 2026-02-25T08:46:15.797 INFO:tasks.rados.rados.0.trial169.stdout:1017: finishing write tid 3 to trial16941936-21 2026-02-25T08:46:15.797 INFO:tasks.rados.rados.0.trial169.stdout:1017: finishing write tid 4 to trial16941936-21 2026-02-25T08:46:15.797 INFO:tasks.rados.rados.0.trial169.stdout:1017: oid 21 updating version 0 to 98 2026-02-25T08:46:15.797 INFO:tasks.rados.rados.0.trial169.stdout:1017: oid 21 version 98 is already newer than 98 2026-02-25T08:46:15.797 INFO:tasks.rados.rados.0.trial169.stdout:1017: oid 21 version 98 is already newer than 98 2026-02-25T08:46:15.797 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 21 v 98 (ObjNum 486 snap 129 seq_num 486) dirty exists 2026-02-25T08:46:15.798 INFO:tasks.rados.rados.0.trial169.stdout:1017: left oid 21 (ObjNum 486 snap 129 seq_num 486) 2026-02-25T08:46:15.798 INFO:tasks.rados.rados.0.trial169.stdout:1012: finishing write tid 1 to trial16941936-33 2026-02-25T08:46:15.798 INFO:tasks.rados.rados.0.trial169.stdout:1016: expect (ObjNum 401 snap 109 seq_num 401) 2026-02-25T08:46:15.806 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1651012 2026-02-25T08:46:15.840 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 910049 2026-02-25T08:46:15.922 INFO:tasks.rados.rados.0.trial169.stdout:1012: finishing write tid 2 to trial16941936-33 2026-02-25T08:46:15.922 INFO:tasks.rados.rados.0.trial169.stdout:1012: finishing write tid 3 to trial16941936-33 2026-02-25T08:46:15.922 INFO:tasks.rados.rados.0.trial169.stdout:1012: finishing write tid 4 to trial16941936-33 2026-02-25T08:46:15.922 INFO:tasks.rados.rados.0.trial169.stdout:1012: oid 33 updating version 0 to 541 2026-02-25T08:46:15.922 INFO:tasks.rados.rados.0.trial169.stdout:1012: oid 33 version 541 is already newer than 541 2026-02-25T08:46:15.922 INFO:tasks.rados.rados.0.trial169.stdout:1012: oid 33 version 541 is already newer than 539 2026-02-25T08:46:15.922 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 33 v 541 (ObjNum 483 snap 129 seq_num 483) dirty exists 2026-02-25T08:46:15.922 INFO:tasks.rados.rados.0.trial169.stdout:1012: left oid 33 (ObjNum 483 snap 129 seq_num 483) 2026-02-25T08:46:16.068 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:16.146 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:16.143+0000 7f1056ec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:16.148 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:16.146+0000 7f1056ec2640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:16.151 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:16.156 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:46:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:16.192 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:16.228 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:16.231 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:16.231 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:46:16.246 INFO:tasks.rados.rados.0.trial169.stdout:1012: done (7 left) 2026-02-25T08:46:16.247 INFO:tasks.rados.rados.0.trial169.stdout:1013: done (6 left) 2026-02-25T08:46:16.247 INFO:tasks.rados.rados.0.trial169.stdout:1014: done (5 left) 2026-02-25T08:46:16.247 INFO:tasks.rados.rados.0.trial169.stdout:1015: done (4 left) 2026-02-25T08:46:16.247 INFO:tasks.rados.rados.0.trial169.stdout:1016: done (3 left) 2026-02-25T08:46:16.247 INFO:tasks.rados.rados.0.trial169.stdout:1017: done (2 left) 2026-02-25T08:46:16.247 INFO:tasks.rados.rados.0.trial169.stdout:1018: done (1 left) 2026-02-25T08:46:16.247 INFO:tasks.rados.rados.0.trial169.stdout:1019: done (0 left) 2026-02-25T08:46:16.247 INFO:tasks.rados.rados.0.trial169.stdout:1020: snap_create 2026-02-25T08:46:17.248 INFO:tasks.rados.rados.0.trial169.stdout:1020: done (0 left) 2026-02-25T08:46:17.248 INFO:tasks.rados.rados.0.trial169.stdout:1021: delete oid 25 current snap is 131 2026-02-25T08:46:17.295 INFO:tasks.rados.rados.0.trial169.stdout:1021: done (0 left) 2026-02-25T08:46:17.295 INFO:tasks.rados.rados.0.trial169.stdout:1022: write oid 35 current snap is 131 2026-02-25T08:46:17.295 INFO:tasks.rados.rados.0.trial169.stdout:1022: seq_num 488 ranges {601816=568104,1623448=451823} 2026-02-25T08:46:17.299 INFO:tasks.rados.rados.0.trial169.stdout:1022: writing trial16941936-35 from 601816 to 1169920 tid 1 2026-02-25T08:46:17.303 INFO:tasks.rados.rados.0.trial169.stdout:1022: writing trial16941936-35 from 1623448 to 2075271 tid 2 2026-02-25T08:46:17.303 INFO:tasks.rados.rados.0.trial169.stdout:1023: read oid 12 snap -1 2026-02-25T08:46:17.303 INFO:tasks.rados.rados.0.trial169.stdout:1023: expect (ObjNum 346 snap 95 seq_num 346) 2026-02-25T08:46:17.303 INFO:tasks.rados.rados.0.trial169.stdout:1024: read oid 42 snap -1 2026-02-25T08:46:17.303 INFO:tasks.rados.rados.0.trial169.stdout:1024: expect (ObjNum 324 snap 87 seq_num 324) 2026-02-25T08:46:17.304 INFO:tasks.rados.rados.0.trial169.stdout:1025: read oid 31 snap 128 2026-02-25T08:46:17.304 INFO:tasks.rados.rados.0.trial169.stdout:1025: expect deleted 2026-02-25T08:46:17.304 INFO:tasks.rados.rados.0.trial169.stdout:1026: delete oid 20 current snap is 131 2026-02-25T08:46:17.307 INFO:tasks.rados.rados.0.trial169.stdout:1026: done (4 left) 2026-02-25T08:46:17.307 INFO:tasks.rados.rados.0.trial169.stdout:1027: delete oid 2 current snap is 131 2026-02-25T08:46:17.333 INFO:tasks.rados.rados.0.trial169.stdout:1022: finishing write tid 1 to trial16941936-35 2026-02-25T08:46:17.333 INFO:tasks.rados.rados.0.trial169.stdout:1024: expect (ObjNum 324 snap 87 seq_num 324) 2026-02-25T08:46:17.441 INFO:tasks.rados.rados.0.trial169.stdout:1024: done (4 left) 2026-02-25T08:46:17.441 INFO:tasks.rados.rados.0.trial169.stdout:1025: done (3 left) 2026-02-25T08:46:17.441 INFO:tasks.rados.rados.0.trial169.stdout:1027: done (2 left) 2026-02-25T08:46:17.441 INFO:tasks.rados.rados.0.trial169.stdout:1028: write (excl) oid 16 current snap is 131 2026-02-25T08:46:17.441 INFO:tasks.rados.rados.0.trial169.stdout:1022: finishing write tid 2 to trial16941936-35 2026-02-25T08:46:17.441 INFO:tasks.rados.rados.0.trial169.stdout:1022: finishing write tid 3 to trial16941936-35 2026-02-25T08:46:17.441 INFO:tasks.rados.rados.0.trial169.stdout:1028: seq_num 491 ranges {652237=793851,2198894=53343} 2026-02-25T08:46:17.446 INFO:tasks.rados.rados.0.trial169.stdout:1028: writing trial16941936-16 from 652237 to 1446088 tid 1 2026-02-25T08:46:17.449 INFO:tasks.rados.rados.0.trial169.stdout:1028: writing trial16941936-16 from 2198894 to 2252237 tid 2 2026-02-25T08:46:17.449 INFO:tasks.rados.rados.0.trial169.stdout:1029: snap_remove snap 122 2026-02-25T08:46:18.162 INFO:tasks.rados.rados.0.trial169.stdout:1029: done (3 left) 2026-02-25T08:46:18.162 INFO:tasks.rados.rados.0.trial169.stdout:1030: write oid 19 current snap is 131 2026-02-25T08:46:18.162 INFO:tasks.rados.rados.0.trial169.stdout:1022: finishing write tid 4 to trial16941936-35 2026-02-25T08:46:18.162 INFO:tasks.rados.rados.0.trial169.stdout:1022: oid 35 updating version 0 to 544 2026-02-25T08:46:18.162 INFO:tasks.rados.rados.0.trial169.stdout:1022: oid 35 version 544 is already newer than 544 2026-02-25T08:46:18.162 INFO:tasks.rados.rados.0.trial169.stdout:1022: oid 35 version 544 is already newer than 544 2026-02-25T08:46:18.162 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 35 v 544 (ObjNum 488 snap 131 seq_num 488) dirty exists 2026-02-25T08:46:18.162 INFO:tasks.rados.rados.0.trial169.stdout:1022: left oid 35 (ObjNum 488 snap 131 seq_num 488) 2026-02-25T08:46:18.163 INFO:tasks.rados.rados.0.trial169.stdout:1030: seq_num 492 ranges {669044=469100,1741023=459161} 2026-02-25T08:46:18.166 INFO:tasks.rados.rados.0.trial169.stdout:1030: writing trial16941936-19 from 669044 to 1138144 tid 1 2026-02-25T08:46:18.170 INFO:tasks.rados.rados.0.trial169.stdout:1030: writing trial16941936-19 from 1741023 to 2200184 tid 2 2026-02-25T08:46:18.170 INFO:tasks.rados.rados.0.trial169.stdout:1022: done (3 left) 2026-02-25T08:46:18.170 INFO:tasks.rados.rados.0.trial169.stdout:1031: read oid 37 snap -1 2026-02-25T08:46:18.171 INFO:tasks.rados.rados.0.trial169.stdout:1031: expect (ObjNum 485 snap 129 seq_num 485) 2026-02-25T08:46:18.171 INFO:tasks.rados.rados.0.trial169.stdout:1032: delete oid 1 current snap is 131 2026-02-25T08:46:18.171 INFO:tasks.rados.rados.0.trial169.stdout:1023: expect (ObjNum 346 snap 95 seq_num 346) 2026-02-25T08:46:18.171 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:18.192 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:18.221 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:18.270 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:18.267+0000 7fb72925a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:18.272 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:18.270+0000 7fb72925a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:18.275 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:18.313 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:18.314 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:18.324 INFO:tasks.rados.rados.0.trial169.stdout:1023: done (4 left) 2026-02-25T08:46:18.324 INFO:tasks.rados.rados.0.trial169.stdout:1032: done (3 left) 2026-02-25T08:46:18.324 INFO:tasks.rados.rados.0.trial169.stdout:1033: write (excl) oid 34 current snap is 131 2026-02-25T08:46:18.324 INFO:tasks.rados.rados.0.trial169.stdout:1028: finishing write tid 1 to trial16941936-16 2026-02-25T08:46:18.324 INFO:tasks.rados.rados.0.trial169.stdout:1028: finishing write tid 2 to trial16941936-16 2026-02-25T08:46:18.324 INFO:tasks.rados.rados.0.trial169.stdout:1033: seq_num 494 ranges {702658=619598,1857987=444671} 2026-02-25T08:46:18.329 INFO:tasks.rados.rados.0.trial169.stdout:1033: writing trial16941936-34 from 702658 to 1322256 tid 1 2026-02-25T08:46:18.333 INFO:tasks.rados.rados.0.trial169.stdout:1033: writing trial16941936-34 from 1857987 to 2302658 tid 2 2026-02-25T08:46:18.333 INFO:tasks.rados.rados.0.trial169.stdout:1034: write (excl) oid 38 current snap is 131 2026-02-25T08:46:18.333 INFO:tasks.rados.rados.0.trial169.stdout:1034: seq_num 495 ranges {719465=411200,1632822=686643} 2026-02-25T08:46:18.337 INFO:tasks.rados.rados.0.trial169.stdout:1034: writing trial16941936-38 from 719465 to 1130665 tid 1 2026-02-25T08:46:18.342 INFO:tasks.rados.rados.0.trial169.stdout:1034: writing trial16941936-38 from 1632822 to 2319465 tid 2 2026-02-25T08:46:18.343 INFO:tasks.rados.rados.0.trial169.stdout:1035: write oid 47 current snap is 131 2026-02-25T08:46:18.343 INFO:tasks.rados.rados.0.trial169.stdout:1028: finishing write tid 3 to trial16941936-16 2026-02-25T08:46:18.343 INFO:tasks.rados.rados.0.trial169.stdout:1028: finishing write tid 4 to trial16941936-16 2026-02-25T08:46:18.343 INFO:tasks.rados.rados.0.trial169.stdout:1028: oid 16 updating version 0 to 179 2026-02-25T08:46:18.343 INFO:tasks.rados.rados.0.trial169.stdout:1028: oid 16 version 179 is already newer than 179 2026-02-25T08:46:18.343 INFO:tasks.rados.rados.0.trial169.stdout:1028: oid 16 version 179 is already newer than 177 2026-02-25T08:46:18.343 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 16 v 179 (ObjNum 491 snap 131 seq_num 491) dirty exists 2026-02-25T08:46:18.343 INFO:tasks.rados.rados.0.trial169.stdout:1028: left oid 16 (ObjNum 491 snap 131 seq_num 491) 2026-02-25T08:46:18.344 INFO:tasks.rados.rados.0.trial169.stdout:1035: seq_num 496 ranges {736272=486449,1974951=361321} 2026-02-25T08:46:18.346 INFO:tasks.rados.rados.0.trial169.stdout:1035: writing trial16941936-47 from 736272 to 1222721 tid 1 2026-02-25T08:46:18.351 INFO:tasks.rados.rados.0.trial169.stdout:1035: writing trial16941936-47 from 1974951 to 2336272 tid 2 2026-02-25T08:46:18.351 INFO:tasks.rados.rados.0.trial169.stdout:1028: done (5 left) 2026-02-25T08:46:18.351 INFO:tasks.rados.rados.0.trial169.stdout:1036: write (excl) oid 49 current snap is 131 2026-02-25T08:46:18.351 INFO:tasks.rados.rados.0.trial169.stdout:1036: seq_num 497 ranges {753079=561698,2033433=319646} 2026-02-25T08:46:18.355 INFO:tasks.rados.rados.0.trial169.stdout:1036: writing trial16941936-49 from 753079 to 1314777 tid 1 2026-02-25T08:46:18.360 INFO:tasks.rados.rados.0.trial169.stdout:1036: writing trial16941936-49 from 2033433 to 2353079 tid 2 2026-02-25T08:46:18.360 INFO:tasks.rados.rados.0.trial169.stdout:1037: snap_create 2026-02-25T08:46:18.360 INFO:tasks.rados.rados.0.trial169.stdout:1031: expect (ObjNum 485 snap 129 seq_num 485) 2026-02-25T08:46:18.360 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:18.390 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:18.411 INFO:tasks.rados.rados.0.trial169.stdout:1030: finishing write tid 1 to trial16941936-19 2026-02-25T08:46:18.411 INFO:tasks.rados.rados.0.trial169.stdout:1030: finishing write tid 2 to trial16941936-19 2026-02-25T08:46:18.411 INFO:tasks.rados.rados.0.trial169.stdout:1030: finishing write tid 3 to trial16941936-19 2026-02-25T08:46:18.411 INFO:tasks.rados.rados.0.trial169.stdout:1030: finishing write tid 4 to trial16941936-19 2026-02-25T08:46:18.411 INFO:tasks.rados.rados.0.trial169.stdout:1030: oid 19 updating version 0 to 158 2026-02-25T08:46:18.411 INFO:tasks.rados.rados.0.trial169.stdout:1030: oid 19 version 158 is already newer than 158 2026-02-25T08:46:18.411 INFO:tasks.rados.rados.0.trial169.stdout:1030: oid 19 version 158 is already newer than 158 2026-02-25T08:46:18.411 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 19 v 158 (ObjNum 492 snap 131 seq_num 492) dirty exists 2026-02-25T08:46:18.411 INFO:tasks.rados.rados.0.trial169.stdout:1030: left oid 19 (ObjNum 492 snap 131 seq_num 492) 2026-02-25T08:46:18.411 INFO:tasks.rados.rados.0.trial169.stdout:1033: finishing write tid 1 to trial16941936-34 2026-02-25T08:46:18.412 INFO:tasks.rados.rados.0.trial169.stdout:1034: finishing write tid 1 to trial16941936-38 2026-02-25T08:46:18.412 INFO:tasks.rados.rados.0.trial169.stdout:1036: finishing write tid 1 to trial16941936-49 2026-02-25T08:46:18.412 INFO:tasks.rados.rados.0.trial169.stdout:1033: finishing write tid 2 to trial16941936-34 2026-02-25T08:46:18.412 INFO:tasks.rados.rados.0.trial169.stdout:1034: finishing write tid 2 to trial16941936-38 2026-02-25T08:46:18.412 INFO:tasks.rados.rados.0.trial169.stdout:1036: finishing write tid 2 to trial16941936-49 2026-02-25T08:46:18.412 INFO:tasks.rados.rados.0.trial169.stdout:1036: finishing write tid 3 to trial16941936-49 2026-02-25T08:46:18.412 INFO:tasks.rados.rados.0.trial169.stdout:1036: finishing write tid 4 to trial16941936-49 2026-02-25T08:46:18.412 INFO:tasks.rados.rados.0.trial169.stdout:1036: oid 49 updating version 0 to 452 2026-02-25T08:46:18.412 INFO:tasks.rados.rados.0.trial169.stdout:1036: oid 49 updating version 452 to 454 2026-02-25T08:46:18.412 INFO:tasks.rados.rados.0.trial169.stdout:1036: oid 49 version 454 is already newer than 454 2026-02-25T08:46:18.412 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 49 v 454 (ObjNum 497 snap 131 seq_num 497) dirty exists 2026-02-25T08:46:18.412 INFO:tasks.rados.rados.0.trial169.stdout:1036: left oid 49 (ObjNum 497 snap 131 seq_num 497) 2026-02-25T08:46:18.413 INFO:tasks.rados.rados.0.trial169.stdout:1034: finishing write tid 3 to trial16941936-38 2026-02-25T08:46:18.413 INFO:tasks.rados.rados.0.trial169.stdout:1033: finishing write tid 3 to trial16941936-34 2026-02-25T08:46:18.413 INFO:tasks.rados.rados.0.trial169.stdout:1034: finishing write tid 4 to trial16941936-38 2026-02-25T08:46:18.413 INFO:tasks.rados.rados.0.trial169.stdout:1034: oid 38 updating version 0 to 104 2026-02-25T08:46:18.413 INFO:tasks.rados.rados.0.trial169.stdout:1034: oid 38 updating version 104 to 106 2026-02-25T08:46:18.413 INFO:tasks.rados.rados.0.trial169.stdout:1034: oid 38 version 106 is already newer than 106 2026-02-25T08:46:18.413 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 38 v 106 (ObjNum 495 snap 131 seq_num 495) dirty exists 2026-02-25T08:46:18.413 INFO:tasks.rados.rados.0.trial169.stdout:1034: left oid 38 (ObjNum 495 snap 131 seq_num 495) 2026-02-25T08:46:18.413 INFO:tasks.rados.rados.0.trial169.stdout:1033: finishing write tid 4 to trial16941936-34 2026-02-25T08:46:18.413 INFO:tasks.rados.rados.0.trial169.stdout:1033: oid 34 updating version 0 to 159 2026-02-25T08:46:18.413 INFO:tasks.rados.rados.0.trial169.stdout:1033: oid 34 updating version 159 to 161 2026-02-25T08:46:18.413 INFO:tasks.rados.rados.0.trial169.stdout:1033: oid 34 version 161 is already newer than 161 2026-02-25T08:46:18.413 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 34 v 161 (ObjNum 494 snap 131 seq_num 494) dirty exists 2026-02-25T08:46:18.414 INFO:tasks.rados.rados.0.trial169.stdout:1033: left oid 34 (ObjNum 494 snap 131 seq_num 494) 2026-02-25T08:46:18.414 INFO:tasks.rados.rados.0.trial169.stdout:1035: finishing write tid 1 to trial16941936-47 2026-02-25T08:46:18.414 INFO:tasks.rados.rados.0.trial169.stdout:1035: finishing write tid 2 to trial16941936-47 2026-02-25T08:46:18.414 INFO:tasks.rados.rados.0.trial169.stdout:1035: finishing write tid 3 to trial16941936-47 2026-02-25T08:46:18.415 INFO:tasks.rados.rados.0.trial169.stdout:1035: finishing write tid 4 to trial16941936-47 2026-02-25T08:46:18.415 INFO:tasks.rados.rados.0.trial169.stdout:1035: oid 47 updating version 0 to 110 2026-02-25T08:46:18.415 INFO:tasks.rados.rados.0.trial169.stdout:1035: oid 47 version 110 is already newer than 110 2026-02-25T08:46:18.415 INFO:tasks.rados.rados.0.trial169.stdout:1035: oid 47 version 110 is already newer than 110 2026-02-25T08:46:18.415 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 47 v 110 (ObjNum 496 snap 131 seq_num 496) dirty exists 2026-02-25T08:46:18.415 INFO:tasks.rados.rados.0.trial169.stdout:1035: left oid 47 (ObjNum 496 snap 131 seq_num 496) 2026-02-25T08:46:19.192 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~253s 2026-02-25T08:46:19.257 INFO:tasks.rados.rados.0.trial169.stdout:1030: done (6 left) 2026-02-25T08:46:19.257 INFO:tasks.rados.rados.0.trial169.stdout:1031: done (5 left) 2026-02-25T08:46:19.257 INFO:tasks.rados.rados.0.trial169.stdout:1033: done (4 left) 2026-02-25T08:46:19.257 INFO:tasks.rados.rados.0.trial169.stdout:1034: done (3 left) 2026-02-25T08:46:19.257 INFO:tasks.rados.rados.0.trial169.stdout:1035: done (2 left) 2026-02-25T08:46:19.258 INFO:tasks.rados.rados.0.trial169.stdout:1036: done (1 left) 2026-02-25T08:46:19.258 INFO:tasks.rados.rados.0.trial169.stdout:1037: done (0 left) 2026-02-25T08:46:19.258 INFO:tasks.rados.rados.0.trial169.stdout:1038: write oid 43 current snap is 132 2026-02-25T08:46:19.258 INFO:tasks.rados.rados.0.trial169.stdout:1038: seq_num 498 ranges {769886=636947,2091915=277971} 2026-02-25T08:46:19.262 INFO:tasks.rados.rados.0.trial169.stdout:1038: writing trial16941936-43 from 769886 to 1406833 tid 1 2026-02-25T08:46:19.266 INFO:tasks.rados.rados.0.trial169.stdout:1038: writing trial16941936-43 from 2091915 to 2369886 tid 2 2026-02-25T08:46:19.266 INFO:tasks.rados.rados.0.trial169.stdout:1039: snap_create 2026-02-25T08:46:19.302 INFO:tasks.rados.rados.0.trial169.stdout:1038: finishing write tid 1 to trial16941936-43 2026-02-25T08:46:19.309 INFO:tasks.rados.rados.0.trial169.stdout:1038: finishing write tid 2 to trial16941936-43 2026-02-25T08:46:19.314 INFO:tasks.rados.rados.0.trial169.stdout:1038: finishing write tid 3 to trial16941936-43 2026-02-25T08:46:19.316 INFO:tasks.rados.rados.0.trial169.stdout:1038: finishing write tid 4 to trial16941936-43 2026-02-25T08:46:19.316 INFO:tasks.rados.rados.0.trial169.stdout:1038: oid 43 updating version 0 to 101 2026-02-25T08:46:19.316 INFO:tasks.rados.rados.0.trial169.stdout:1038: oid 43 version 101 is already newer than 101 2026-02-25T08:46:19.316 INFO:tasks.rados.rados.0.trial169.stdout:1038: oid 43 version 101 is already newer than 101 2026-02-25T08:46:19.317 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 43 v 101 (ObjNum 498 snap 132 seq_num 498) dirty exists 2026-02-25T08:46:19.317 INFO:tasks.rados.rados.0.trial169.stdout:1038: left oid 43 (ObjNum 498 snap 132 seq_num 498) 2026-02-25T08:46:20.260 INFO:tasks.rados.rados.0.trial169.stdout:1038: done (1 left) 2026-02-25T08:46:20.260 INFO:tasks.rados.rados.0.trial169.stdout:1039: done (0 left) 2026-02-25T08:46:20.261 INFO:tasks.rados.rados.0.trial169.stdout:1040: read oid 22 snap -1 2026-02-25T08:46:20.261 INFO:tasks.rados.rados.0.trial169.stdout:1040: expect (ObjNum 433 snap 115 seq_num 433) 2026-02-25T08:46:20.261 INFO:tasks.rados.rados.0.trial169.stdout:1041: read oid 14 snap 128 2026-02-25T08:46:20.261 INFO:tasks.rados.rados.0.trial169.stdout:1041: expect (ObjNum 454 snap 122 seq_num 454) 2026-02-25T08:46:20.261 INFO:tasks.rados.rados.0.trial169.stdout:1042: write (excl) oid 6 current snap is 133 2026-02-25T08:46:20.261 INFO:tasks.rados.rados.0.trial169.stdout:1042: seq_num 499 ranges {786693=712196,2150397=236296} 2026-02-25T08:46:20.266 INFO:tasks.rados.rados.0.trial169.stdout:1042: writing trial16941936-6 from 786693 to 1498889 tid 1 2026-02-25T08:46:20.270 INFO:tasks.rados.rados.0.trial169.stdout:1042: writing trial16941936-6 from 2150397 to 2386693 tid 2 2026-02-25T08:46:20.270 INFO:tasks.rados.rados.0.trial169.stdout:1043: delete oid 44 current snap is 133 2026-02-25T08:46:20.288 INFO:tasks.rados.rados.0.trial169.stdout:1042: finishing write tid 1 to trial16941936-6 2026-02-25T08:46:20.289 INFO:tasks.rados.rados.0.trial169.stdout:1040: expect (ObjNum 433 snap 115 seq_num 433) 2026-02-25T08:46:20.301 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2076902 2026-02-25T08:46:20.315 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:20.317 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:20.356 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 834092 2026-02-25T08:46:20.386 INFO:tasks.rados.rados.0.trial169.stdout:1042: finishing write tid 2 to trial16941936-6 2026-02-25T08:46:20.386 INFO:tasks.rados.rados.0.trial169.stdout:1042: finishing write tid 3 to trial16941936-6 2026-02-25T08:46:20.387 INFO:tasks.rados.rados.0.trial169.stdout:1042: finishing write tid 4 to trial16941936-6 2026-02-25T08:46:20.387 INFO:tasks.rados.rados.0.trial169.stdout:1042: oid 6 updating version 0 to 104 2026-02-25T08:46:20.387 INFO:tasks.rados.rados.0.trial169.stdout:1042: oid 6 version 104 is already newer than 102 2026-02-25T08:46:20.387 INFO:tasks.rados.rados.0.trial169.stdout:1042: oid 6 version 104 is already newer than 104 2026-02-25T08:46:20.387 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 6 v 104 (ObjNum 499 snap 133 seq_num 499) dirty exists 2026-02-25T08:46:20.387 INFO:tasks.rados.rados.0.trial169.stdout:1042: left oid 6 (ObjNum 499 snap 133 seq_num 499) 2026-02-25T08:46:20.387 INFO:tasks.rados.rados.0.trial169.stdout:1041: expect (ObjNum 454 snap 122 seq_num 454) 2026-02-25T08:46:20.392 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:20.389+0000 7f1a82793640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:20.394 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:20.393+0000 7f1a82793640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:20.398 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:20.439 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:20.440 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:20.449 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 3439854 2026-02-25T08:46:20.450 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:20.610 INFO:tasks.rados.rados.0.trial169.stdout:1040: done (3 left) 2026-02-25T08:46:20.611 INFO:tasks.rados.rados.0.trial169.stdout:1041: done (2 left) 2026-02-25T08:46:20.611 INFO:tasks.rados.rados.0.trial169.stdout:1042: done (1 left) 2026-02-25T08:46:20.611 INFO:tasks.rados.rados.0.trial169.stdout:1043: done (0 left) 2026-02-25T08:46:20.611 INFO:tasks.rados.rados.0.trial169.stdout:1044: read oid 36 snap -1 2026-02-25T08:46:20.611 INFO:tasks.rados.rados.0.trial169.stdout:1044: expect (ObjNum 450 snap 121 seq_num 450) 2026-02-25T08:46:20.611 INFO:tasks.rados.rados.0.trial169.stdout:1045: snap_remove snap 119 2026-02-25T08:46:21.233 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:46:21.262 INFO:tasks.rados.rados.0.trial169.stdout:1045: done (1 left) 2026-02-25T08:46:21.262 INFO:tasks.rados.rados.0.trial169.stdout:1046: read oid 29 snap 129 2026-02-25T08:46:21.263 INFO:tasks.rados.rados.0.trial169.stdout:1046: expect deleted 2026-02-25T08:46:21.263 INFO:tasks.rados.rados.0.trial169.stdout:1047: read oid 21 snap -1 2026-02-25T08:46:21.263 INFO:tasks.rados.rados.0.trial169.stdout:1047: expect (ObjNum 486 snap 129 seq_num 486) 2026-02-25T08:46:21.263 INFO:tasks.rados.rados.0.trial169.stdout:1048: read oid 1 snap 130 2026-02-25T08:46:21.263 INFO:tasks.rados.rados.0.trial169.stdout:1048: expect (ObjNum 381 snap 102 seq_num 381) 2026-02-25T08:46:21.263 INFO:tasks.rados.rados.0.trial169.stdout:1049: snap_create 2026-02-25T08:46:21.263 INFO:tasks.rados.rados.0.trial169.stdout:1044: expect (ObjNum 450 snap 121 seq_num 450) 2026-02-25T08:46:21.278 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2298839 2026-02-25T08:46:21.308 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:21.311 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:21.312 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:46:21.337 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 662562 2026-02-25T08:46:21.376 INFO:tasks.rados.rados.0.trial169.stdout:1048: expect (ObjNum 381 snap 102 seq_num 381) 2026-02-25T08:46:21.378 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 641493 2026-02-25T08:46:21.450 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1176013 2026-02-25T08:46:21.475 INFO:tasks.rados.rados.0.trial169.stdout:1047: expect (ObjNum 486 snap 129 seq_num 486) 2026-02-25T08:46:22.264 INFO:tasks.rados.rados.0.trial169.stdout:1044: done (4 left) 2026-02-25T08:46:22.264 INFO:tasks.rados.rados.0.trial169.stdout:1046: done (3 left) 2026-02-25T08:46:22.264 INFO:tasks.rados.rados.0.trial169.stdout:1047: done (2 left) 2026-02-25T08:46:22.264 INFO:tasks.rados.rados.0.trial169.stdout:1048: done (1 left) 2026-02-25T08:46:22.265 INFO:tasks.rados.rados.0.trial169.stdout:1049: done (0 left) 2026-02-25T08:46:22.265 INFO:tasks.rados.rados.0.trial169.stdout:1050: read oid 43 snap 130 2026-02-25T08:46:22.265 INFO:tasks.rados.rados.0.trial169.stdout:1050: expect (ObjNum 436 snap 116 seq_num 436) 2026-02-25T08:46:22.265 INFO:tasks.rados.rados.0.trial169.stdout:1051: read oid 5 snap -1 2026-02-25T08:46:22.265 INFO:tasks.rados.rados.0.trial169.stdout:1051: expect (ObjNum 452 snap 122 seq_num 452) 2026-02-25T08:46:22.265 INFO:tasks.rados.rados.0.trial169.stdout:1052: read oid 23 snap -1 2026-02-25T08:46:22.265 INFO:tasks.rados.rados.0.trial169.stdout:1052: expect (ObjNum 455 snap 122 seq_num 455) 2026-02-25T08:46:22.265 INFO:tasks.rados.rados.0.trial169.stdout:1053: snap_create 2026-02-25T08:46:22.296 INFO:tasks.rados.rados.0.trial169.stdout:1052: expect (ObjNum 455 snap 122 seq_num 455) 2026-02-25T08:46:22.324 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2847616 2026-02-25T08:46:22.339 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1020020 2026-02-25T08:46:22.361 INFO:tasks.rados.rados.0.trial169.stdout:1050: expect (ObjNum 436 snap 116 seq_num 436) 2026-02-25T08:46:22.361 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:22.440 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:22.518 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:22.515+0000 7f5f1868b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:22.520 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:22.518+0000 7f5f1868b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:22.523 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:22.529 INFO:tasks.rados.rados.0.trial169.stdout:1051: expect (ObjNum 452 snap 122 seq_num 452) 2026-02-25T08:46:22.548 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2228326 2026-02-25T08:46:22.564 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:22.565 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:22.565 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:22.612 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:23.268 INFO:tasks.rados.rados.0.trial169.stdout:1050: done (3 left) 2026-02-25T08:46:23.269 INFO:tasks.rados.rados.0.trial169.stdout:1051: done (2 left) 2026-02-25T08:46:23.269 INFO:tasks.rados.rados.0.trial169.stdout:1052: done (1 left) 2026-02-25T08:46:23.269 INFO:tasks.rados.rados.0.trial169.stdout:1053: done (0 left) 2026-02-25T08:46:23.269 INFO:tasks.rados.rados.0.trial169.stdout:1054: read oid 33 snap 132 2026-02-25T08:46:23.269 INFO:tasks.rados.rados.0.trial169.stdout:1054: expect (ObjNum 483 snap 129 seq_num 483) 2026-02-25T08:46:23.269 INFO:tasks.rados.rados.0.trial169.stdout:1055: write (excl) oid 14 current snap is 135 2026-02-25T08:46:23.269 INFO:tasks.rados.rados.0.trial169.stdout:1055: seq_num 501 ranges {420307=462694,1351008=617495} 2026-02-25T08:46:23.271 INFO:tasks.rados.rados.0.trial169.stdout:1055: writing trial16941936-14 from 420307 to 883001 tid 1 2026-02-25T08:46:23.276 INFO:tasks.rados.rados.0.trial169.stdout:1055: writing trial16941936-14 from 1351008 to 1968503 tid 2 2026-02-25T08:46:23.276 INFO:tasks.rados.rados.0.trial169.stdout:1056: write (excl) oid 27 current snap is 135 2026-02-25T08:46:23.276 INFO:tasks.rados.rados.0.trial169.stdout:1056: seq_num 502 ranges {437114=654296,1525843=477506} 2026-02-25T08:46:23.280 INFO:tasks.rados.rados.0.trial169.stdout:1056: writing trial16941936-27 from 437114 to 1091410 tid 1 2026-02-25T08:46:23.284 INFO:tasks.rados.rados.0.trial169.stdout:1056: writing trial16941936-27 from 1525843 to 2003349 tid 2 2026-02-25T08:46:23.284 INFO:tasks.rados.rados.0.trial169.stdout:1057: write oid 15 current snap is 135 2026-02-25T08:46:23.284 INFO:tasks.rados.rados.0.trial169.stdout:1057: seq_num 503 ranges {453921=729545,1584325=621164} 2026-02-25T08:46:23.289 INFO:tasks.rados.rados.0.trial169.stdout:1057: writing trial16941936-15 from 453921 to 1183466 tid 1 2026-02-25T08:46:23.294 INFO:tasks.rados.rados.0.trial169.stdout:1057: writing trial16941936-15 from 1584325 to 2205489 tid 2 2026-02-25T08:46:23.294 INFO:tasks.rados.rados.0.trial169.stdout:1058: write oid 9 current snap is 135 2026-02-25T08:46:23.294 INFO:tasks.rados.rados.0.trial169.stdout:1058: seq_num 504 ranges {470728=404794,1526454=481175} 2026-02-25T08:46:23.297 INFO:tasks.rados.rados.0.trial169.stdout:1058: writing trial16941936-9 from 470728 to 875522 tid 1 2026-02-25T08:46:23.301 INFO:tasks.rados.rados.0.trial169.stdout:1058: writing trial16941936-9 from 1526454 to 2007629 tid 2 2026-02-25T08:46:23.301 INFO:tasks.rados.rados.0.trial169.stdout:1059: delete oid 39 current snap is 135 2026-02-25T08:46:23.313 INFO:tasks.rados.rados.0.trial169.stdout:1054: expect (ObjNum 483 snap 129 seq_num 483) 2026-02-25T08:46:23.314 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:23.359 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1119708 2026-02-25T08:46:23.397 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1309145 2026-02-25T08:46:23.419 INFO:tasks.rados.rados.0.trial169.stdout:1058: finishing write tid 1 to trial16941936-9 2026-02-25T08:46:23.419 INFO:tasks.rados.rados.0.trial169.stdout:1055: finishing write tid 1 to trial16941936-14 2026-02-25T08:46:23.419 INFO:tasks.rados.rados.0.trial169.stdout:1058: finishing write tid 2 to trial16941936-9 2026-02-25T08:46:23.419 INFO:tasks.rados.rados.0.trial169.stdout:1054: done (5 left) 2026-02-25T08:46:23.420 INFO:tasks.rados.rados.0.trial169.stdout:1059: done (4 left) 2026-02-25T08:46:23.420 INFO:tasks.rados.rados.0.trial169.stdout:1060: snap_create 2026-02-25T08:46:23.420 INFO:tasks.rados.rados.0.trial169.stdout:1058: finishing write tid 3 to trial16941936-9 2026-02-25T08:46:23.420 INFO:tasks.rados.rados.0.trial169.stdout:1058: finishing write tid 4 to trial16941936-9 2026-02-25T08:46:23.420 INFO:tasks.rados.rados.0.trial169.stdout:1058: oid 9 updating version 0 to 164 2026-02-25T08:46:23.420 INFO:tasks.rados.rados.0.trial169.stdout:1058: oid 9 version 164 is already newer than 164 2026-02-25T08:46:23.420 INFO:tasks.rados.rados.0.trial169.stdout:1058: oid 9 version 164 is already newer than 164 2026-02-25T08:46:23.420 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 9 v 164 (ObjNum 504 snap 135 seq_num 504) dirty exists 2026-02-25T08:46:23.420 INFO:tasks.rados.rados.0.trial169.stdout:1058: left oid 9 (ObjNum 504 snap 135 seq_num 504) 2026-02-25T08:46:23.420 INFO:tasks.rados.rados.0.trial169.stdout:1055: finishing write tid 2 to trial16941936-14 2026-02-25T08:46:23.420 INFO:tasks.rados.rados.0.trial169.stdout:1056: finishing write tid 1 to trial16941936-27 2026-02-25T08:46:23.420 INFO:tasks.rados.rados.0.trial169.stdout:1055: finishing write tid 3 to trial16941936-14 2026-02-25T08:46:23.421 INFO:tasks.rados.rados.0.trial169.stdout:1055: finishing write tid 4 to trial16941936-14 2026-02-25T08:46:23.421 INFO:tasks.rados.rados.0.trial169.stdout:1055: oid 14 updating version 0 to 169 2026-02-25T08:46:23.421 INFO:tasks.rados.rados.0.trial169.stdout:1055: oid 14 version 169 is already newer than 167 2026-02-25T08:46:23.421 INFO:tasks.rados.rados.0.trial169.stdout:1055: oid 14 version 169 is already newer than 169 2026-02-25T08:46:23.421 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 14 v 169 (ObjNum 501 snap 135 seq_num 501) dirty exists 2026-02-25T08:46:23.421 INFO:tasks.rados.rados.0.trial169.stdout:1055: left oid 14 (ObjNum 501 snap 135 seq_num 501) 2026-02-25T08:46:23.421 INFO:tasks.rados.rados.0.trial169.stdout:1056: finishing write tid 2 to trial16941936-27 2026-02-25T08:46:23.421 INFO:tasks.rados.rados.0.trial169.stdout:1056: finishing write tid 3 to trial16941936-27 2026-02-25T08:46:23.421 INFO:tasks.rados.rados.0.trial169.stdout:1056: finishing write tid 4 to trial16941936-27 2026-02-25T08:46:23.421 INFO:tasks.rados.rados.0.trial169.stdout:1056: oid 27 updating version 0 to 108 2026-02-25T08:46:23.421 INFO:tasks.rados.rados.0.trial169.stdout:1056: oid 27 version 108 is already newer than 106 2026-02-25T08:46:23.421 INFO:tasks.rados.rados.0.trial169.stdout:1056: oid 27 version 108 is already newer than 108 2026-02-25T08:46:23.422 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 27 v 108 (ObjNum 502 snap 135 seq_num 502) dirty exists 2026-02-25T08:46:23.422 INFO:tasks.rados.rados.0.trial169.stdout:1056: left oid 27 (ObjNum 502 snap 135 seq_num 502) 2026-02-25T08:46:23.422 INFO:tasks.rados.rados.0.trial169.stdout:1057: finishing write tid 1 to trial16941936-15 2026-02-25T08:46:23.422 INFO:tasks.rados.rados.0.trial169.stdout:1057: finishing write tid 2 to trial16941936-15 2026-02-25T08:46:23.422 INFO:tasks.rados.rados.0.trial169.stdout:1057: finishing write tid 3 to trial16941936-15 2026-02-25T08:46:23.422 INFO:tasks.rados.rados.0.trial169.stdout:1057: finishing write tid 4 to trial16941936-15 2026-02-25T08:46:23.422 INFO:tasks.rados.rados.0.trial169.stdout:1057: oid 15 updating version 0 to 189 2026-02-25T08:46:23.422 INFO:tasks.rados.rados.0.trial169.stdout:1057: oid 15 version 189 is already newer than 189 2026-02-25T08:46:23.422 INFO:tasks.rados.rados.0.trial169.stdout:1057: oid 15 version 189 is already newer than 189 2026-02-25T08:46:23.422 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 15 v 189 (ObjNum 503 snap 135 seq_num 503) dirty exists 2026-02-25T08:46:23.422 INFO:tasks.rados.rados.0.trial169.stdout:1057: left oid 15 (ObjNum 503 snap 135 seq_num 503) 2026-02-25T08:46:24.274 INFO:tasks.rados.rados.0.trial169.stdout:1055: done (4 left) 2026-02-25T08:46:24.274 INFO:tasks.rados.rados.0.trial169.stdout:1056: done (3 left) 2026-02-25T08:46:24.275 INFO:tasks.rados.rados.0.trial169.stdout:1057: done (2 left) 2026-02-25T08:46:24.275 INFO:tasks.rados.rados.0.trial169.stdout:1058: done (1 left) 2026-02-25T08:46:24.275 INFO:tasks.rados.rados.0.trial169.stdout:1060: done (0 left) 2026-02-25T08:46:24.275 INFO:tasks.rados.rados.0.trial169.stdout:1061: write (excl) oid 45 current snap is 136 2026-02-25T08:46:24.275 INFO:tasks.rados.rados.0.trial169.stdout:1061: seq_num 506 ranges {504342=555292,1643418=484844} 2026-02-25T08:46:24.278 INFO:tasks.rados.rados.0.trial169.stdout:1061: writing trial16941936-45 from 504342 to 1059634 tid 1 2026-02-25T08:46:24.283 INFO:tasks.rados.rados.0.trial169.stdout:1061: writing trial16941936-45 from 1643418 to 2128262 tid 2 2026-02-25T08:46:24.283 INFO:tasks.rados.rados.0.trial169.stdout:1062: snap_remove snap 128 2026-02-25T08:46:24.567 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:24.645 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:24.642+0000 7fb53301a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:24.647 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:24.646+0000 7fb53301a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:24.651 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:24.688 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:24.688 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:24.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~259s 2026-02-25T08:46:25.277 INFO:tasks.rados.rados.0.trial169.stdout:1062: done (1 left) 2026-02-25T08:46:25.277 INFO:tasks.rados.rados.0.trial169.stdout:1063: write oid 31 current snap is 136 2026-02-25T08:46:25.278 INFO:tasks.rados.rados.0.trial169.stdout:1061: finishing write tid 1 to trial16941936-45 2026-02-25T08:46:25.278 INFO:tasks.rados.rados.0.trial169.stdout:1063: seq_num 507 ranges {521149=630541,1701900=628502} 2026-02-25T08:46:25.281 INFO:tasks.rados.rados.0.trial169.stdout:1063: writing trial16941936-31 from 521149 to 1151690 tid 1 2026-02-25T08:46:25.286 INFO:tasks.rados.rados.0.trial169.stdout:1063: writing trial16941936-31 from 1701900 to 2330402 tid 2 2026-02-25T08:46:25.286 INFO:tasks.rados.rados.0.trial169.stdout:1064: read oid 40 snap -1 2026-02-25T08:46:25.287 INFO:tasks.rados.rados.0.trial169.stdout:1064: expect deleted 2026-02-25T08:46:25.287 INFO:tasks.rados.rados.0.trial169.stdout:1065: read oid 13 snap -1 2026-02-25T08:46:25.287 INFO:tasks.rados.rados.0.trial169.stdout:1065: expect deleted 2026-02-25T08:46:25.287 INFO:tasks.rados.rados.0.trial169.stdout:1066: snap_create 2026-02-25T08:46:25.287 INFO:tasks.rados.rados.0.trial169.stdout:1061: finishing write tid 2 to trial16941936-45 2026-02-25T08:46:25.287 INFO:tasks.rados.rados.0.trial169.stdout:1061: finishing write tid 3 to trial16941936-45 2026-02-25T08:46:25.287 INFO:tasks.rados.rados.0.trial169.stdout:1061: finishing write tid 4 to trial16941936-45 2026-02-25T08:46:25.287 INFO:tasks.rados.rados.0.trial169.stdout:1061: oid 45 updating version 0 to 173 2026-02-25T08:46:25.287 INFO:tasks.rados.rados.0.trial169.stdout:1061: oid 45 version 173 is already newer than 173 2026-02-25T08:46:25.287 INFO:tasks.rados.rados.0.trial169.stdout:1061: oid 45 version 173 is already newer than 171 2026-02-25T08:46:25.287 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 45 v 173 (ObjNum 506 snap 136 seq_num 506) dirty exists 2026-02-25T08:46:25.287 INFO:tasks.rados.rados.0.trial169.stdout:1061: left oid 45 (ObjNum 506 snap 136 seq_num 506) 2026-02-25T08:46:25.344 INFO:tasks.rados.rados.0.trial169.stdout:1063: finishing write tid 1 to trial16941936-31 2026-02-25T08:46:25.349 INFO:tasks.rados.rados.0.trial169.stdout:1063: finishing write tid 2 to trial16941936-31 2026-02-25T08:46:25.353 INFO:tasks.rados.rados.0.trial169.stdout:1063: finishing write tid 3 to trial16941936-31 2026-02-25T08:46:25.354 INFO:tasks.rados.rados.0.trial169.stdout:1063: finishing write tid 4 to trial16941936-31 2026-02-25T08:46:25.354 INFO:tasks.rados.rados.0.trial169.stdout:1063: oid 31 updating version 0 to 466 2026-02-25T08:46:25.354 INFO:tasks.rados.rados.0.trial169.stdout:1063: oid 31 version 466 is already newer than 466 2026-02-25T08:46:25.354 INFO:tasks.rados.rados.0.trial169.stdout:1063: oid 31 version 466 is already newer than 466 2026-02-25T08:46:25.355 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 31 v 466 (ObjNum 507 snap 136 seq_num 507) dirty exists 2026-02-25T08:46:25.355 INFO:tasks.rados.rados.0.trial169.stdout:1063: left oid 31 (ObjNum 507 snap 136 seq_num 507) 2026-02-25T08:46:26.281 INFO:tasks.rados.rados.0.trial169.stdout:1061: done (4 left) 2026-02-25T08:46:26.282 INFO:tasks.rados.rados.0.trial169.stdout:1063: done (3 left) 2026-02-25T08:46:26.282 INFO:tasks.rados.rados.0.trial169.stdout:1064: done (2 left) 2026-02-25T08:46:26.282 INFO:tasks.rados.rados.0.trial169.stdout:1065: done (1 left) 2026-02-25T08:46:26.282 INFO:tasks.rados.rados.0.trial169.stdout:1066: done (0 left) 2026-02-25T08:46:26.282 INFO:tasks.rados.rados.0.trial169.stdout:1067: write (excl) oid 23 current snap is 137 2026-02-25T08:46:26.282 INFO:tasks.rados.rados.0.trial169.stdout:1067: seq_num 508 ranges {537956=705790,1644029=772160} 2026-02-25T08:46:26.286 INFO:tasks.rados.rados.0.trial169.stdout:1067: writing trial16941936-23 from 537956 to 1243746 tid 1 2026-02-25T08:46:26.291 INFO:tasks.rados.rados.0.trial169.stdout:1067: writing trial16941936-23 from 1644029 to 2416189 tid 2 2026-02-25T08:46:26.291 INFO:tasks.rados.rados.0.trial169.stdout:1068: read oid 33 snap 132 2026-02-25T08:46:26.291 INFO:tasks.rados.rados.0.trial169.stdout:1068: expect (ObjNum 483 snap 129 seq_num 483) 2026-02-25T08:46:26.291 INFO:tasks.rados.rados.0.trial169.stdout:1069: write oid 2 current snap is 137 2026-02-25T08:46:26.291 INFO:tasks.rados.rados.0.trial169.stdout:1069: seq_num 509 ranges {554763=781039,2102511=452252} 2026-02-25T08:46:26.295 INFO:tasks.rados.rados.0.trial169.stdout:1069: writing trial16941936-2 from 554763 to 1335802 tid 1 2026-02-25T08:46:26.301 INFO:tasks.rados.rados.0.trial169.stdout:1069: writing trial16941936-2 from 2102511 to 2554763 tid 2 2026-02-25T08:46:26.301 INFO:tasks.rados.rados.0.trial169.stdout:1070: write oid 5 current snap is 137 2026-02-25T08:46:26.301 INFO:tasks.rados.rados.0.trial169.stdout:1070: seq_num 510 ranges {571570=572641,1877346=694224} 2026-02-25T08:46:26.304 INFO:tasks.rados.rados.0.trial169.stdout:1070: writing trial16941936-5 from 571570 to 1144211 tid 1 2026-02-25T08:46:26.310 INFO:tasks.rados.rados.0.trial169.stdout:1070: writing trial16941936-5 from 1877346 to 2571570 tid 2 2026-02-25T08:46:26.310 INFO:tasks.rados.rados.0.trial169.stdout:1071: write (excl) oid 49 current snap is 137 2026-02-25T08:46:26.310 INFO:tasks.rados.rados.0.trial169.stdout:1071: seq_num 511 ranges {588377=647890,1935828=635840} 2026-02-25T08:46:26.313 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:46:26.315 INFO:tasks.rados.rados.0.trial169.stdout:1071: writing trial16941936-49 from 588377 to 1236267 tid 1 2026-02-25T08:46:26.320 INFO:tasks.rados.rados.0.trial169.stdout:1071: writing trial16941936-49 from 1935828 to 2571668 tid 2 2026-02-25T08:46:26.320 INFO:tasks.rados.rados.0.trial169.stdout:1072: snap_remove snap 130 2026-02-25T08:46:26.386 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:26.389 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:46:26.689 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:26.768 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:26.765+0000 7fb5263dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:26.770 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:26.769+0000 7fb5263dc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:26.774 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:26.814 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:26.815 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:27.286 INFO:tasks.rados.rados.0.trial169.stdout:1072: done (5 left) 2026-02-25T08:46:27.286 INFO:tasks.rados.rados.0.trial169.stdout:1073: delete oid 1 current snap is 137 2026-02-25T08:46:27.286 INFO:tasks.rados.rados.0.trial169.stdout:1069: finishing write tid 1 to trial16941936-2 2026-02-25T08:46:27.286 INFO:tasks.rados.rados.0.trial169.stdout:1069: finishing write tid 2 to trial16941936-2 2026-02-25T08:46:27.286 INFO:tasks.rados.rados.0.trial169.stdout:1069: finishing write tid 3 to trial16941936-2 2026-02-25T08:46:27.286 INFO:tasks.rados.rados.0.trial169.stdout:1069: finishing write tid 4 to trial16941936-2 2026-02-25T08:46:27.287 INFO:tasks.rados.rados.0.trial169.stdout:1069: oid 2 updating version 0 to 103 2026-02-25T08:46:27.287 INFO:tasks.rados.rados.0.trial169.stdout:1069: oid 2 version 103 is already newer than 103 2026-02-25T08:46:27.287 INFO:tasks.rados.rados.0.trial169.stdout:1069: oid 2 version 103 is already newer than 103 2026-02-25T08:46:27.287 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 2 v 103 (ObjNum 509 snap 137 seq_num 509) dirty exists 2026-02-25T08:46:27.287 INFO:tasks.rados.rados.0.trial169.stdout:1069: left oid 2 (ObjNum 509 snap 137 seq_num 509) 2026-02-25T08:46:27.287 INFO:tasks.rados.rados.0.trial169.stdout:1071: finishing write tid 1 to trial16941936-49 2026-02-25T08:46:27.287 INFO:tasks.rados.rados.0.trial169.stdout:1070: finishing write tid 1 to trial16941936-5 2026-02-25T08:46:27.287 INFO:tasks.rados.rados.0.trial169.stdout:1070: finishing write tid 2 to trial16941936-5 2026-02-25T08:46:27.287 INFO:tasks.rados.rados.0.trial169.stdout:1071: finishing write tid 2 to trial16941936-49 2026-02-25T08:46:27.287 INFO:tasks.rados.rados.0.trial169.stdout:1071: finishing write tid 3 to trial16941936-49 2026-02-25T08:46:27.287 INFO:tasks.rados.rados.0.trial169.stdout:1068: expect (ObjNum 483 snap 129 seq_num 483) 2026-02-25T08:46:27.325 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:27.364 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:27.404 INFO:tasks.rados.rados.0.trial169.stdout:1070: finishing write tid 3 to trial16941936-5 2026-02-25T08:46:27.404 INFO:tasks.rados.rados.0.trial169.stdout:1071: finishing write tid 4 to trial16941936-49 2026-02-25T08:46:27.404 INFO:tasks.rados.rados.0.trial169.stdout:1071: oid 49 updating version 0 to 470 2026-02-25T08:46:27.404 INFO:tasks.rados.rados.0.trial169.stdout:1071: oid 49 version 470 is already newer than 470 2026-02-25T08:46:27.404 INFO:tasks.rados.rados.0.trial169.stdout:1071: oid 49 version 470 is already newer than 468 2026-02-25T08:46:27.404 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 49 v 470 (ObjNum 511 snap 137 seq_num 511) dirty exists 2026-02-25T08:46:27.404 INFO:tasks.rados.rados.0.trial169.stdout:1071: left oid 49 (ObjNum 511 snap 137 seq_num 511) 2026-02-25T08:46:27.404 INFO:tasks.rados.rados.0.trial169.stdout:1070: finishing write tid 4 to trial16941936-5 2026-02-25T08:46:27.404 INFO:tasks.rados.rados.0.trial169.stdout:1070: oid 5 updating version 0 to 230 2026-02-25T08:46:27.404 INFO:tasks.rados.rados.0.trial169.stdout:1070: oid 5 version 230 is already newer than 230 2026-02-25T08:46:27.405 INFO:tasks.rados.rados.0.trial169.stdout:1070: oid 5 version 230 is already newer than 230 2026-02-25T08:46:27.405 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 5 v 230 (ObjNum 510 snap 137 seq_num 510) dirty exists 2026-02-25T08:46:27.405 INFO:tasks.rados.rados.0.trial169.stdout:1070: left oid 5 (ObjNum 510 snap 137 seq_num 510) 2026-02-25T08:46:27.405 INFO:tasks.rados.rados.0.trial169.stdout:1067: finishing write tid 1 to trial16941936-23 2026-02-25T08:46:27.405 INFO:tasks.rados.rados.0.trial169.stdout:1067: finishing write tid 2 to trial16941936-23 2026-02-25T08:46:27.405 INFO:tasks.rados.rados.0.trial169.stdout:1067: finishing write tid 3 to trial16941936-23 2026-02-25T08:46:27.405 INFO:tasks.rados.rados.0.trial169.stdout:1068: done (5 left) 2026-02-25T08:46:27.405 INFO:tasks.rados.rados.0.trial169.stdout:1069: done (4 left) 2026-02-25T08:46:27.405 INFO:tasks.rados.rados.0.trial169.stdout:1070: done (3 left) 2026-02-25T08:46:27.405 INFO:tasks.rados.rados.0.trial169.stdout:1071: done (2 left) 2026-02-25T08:46:27.405 INFO:tasks.rados.rados.0.trial169.stdout:1073: done (1 left) 2026-02-25T08:46:27.405 INFO:tasks.rados.rados.0.trial169.stdout:1074: read oid 14 snap -1 2026-02-25T08:46:27.405 INFO:tasks.rados.rados.0.trial169.stdout:1074: expect (ObjNum 501 snap 135 seq_num 501) 2026-02-25T08:46:27.406 INFO:tasks.rados.rados.0.trial169.stdout:1075: read oid 2 snap -1 2026-02-25T08:46:27.406 INFO:tasks.rados.rados.0.trial169.stdout:1075: expect (ObjNum 509 snap 137 seq_num 509) 2026-02-25T08:46:27.406 INFO:tasks.rados.rados.0.trial169.stdout:1076: snap_create 2026-02-25T08:46:27.406 INFO:tasks.rados.rados.0.trial169.stdout:1067: finishing write tid 4 to trial16941936-23 2026-02-25T08:46:27.406 INFO:tasks.rados.rados.0.trial169.stdout:1067: oid 23 updating version 0 to 168 2026-02-25T08:46:27.406 INFO:tasks.rados.rados.0.trial169.stdout:1067: oid 23 version 168 is already newer than 168 2026-02-25T08:46:27.406 INFO:tasks.rados.rados.0.trial169.stdout:1067: oid 23 version 168 is already newer than 166 2026-02-25T08:46:27.406 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 23 v 168 (ObjNum 508 snap 137 seq_num 508) dirty exists 2026-02-25T08:46:27.406 INFO:tasks.rados.rados.0.trial169.stdout:1067: left oid 23 (ObjNum 508 snap 137 seq_num 508) 2026-02-25T08:46:27.419 INFO:tasks.rados.rados.0.trial169.stdout:1075: expect (ObjNum 509 snap 137 seq_num 509) 2026-02-25T08:46:27.482 INFO:tasks.rados.rados.0.trial169.stdout:1074: expect (ObjNum 501 snap 135 seq_num 501) 2026-02-25T08:46:27.482 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:27.536 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1864075 2026-02-25T08:46:28.165 INFO:tasks.rados.rados.0.trial169.stdout:1067: done (3 left) 2026-02-25T08:46:28.165 INFO:tasks.rados.rados.0.trial169.stdout:1074: done (2 left) 2026-02-25T08:46:28.165 INFO:tasks.rados.rados.0.trial169.stdout:1075: done (1 left) 2026-02-25T08:46:28.165 INFO:tasks.rados.rados.0.trial169.stdout:1076: done (0 left) 2026-02-25T08:46:28.165 INFO:tasks.rados.rados.0.trial169.stdout:1077: write (excl) oid 31 current snap is 138 2026-02-25T08:46:28.165 INFO:tasks.rados.rados.0.trial169.stdout:1077: seq_num 513 ranges {621991=798388,1936439=639509} 2026-02-25T08:46:28.170 INFO:tasks.rados.rados.0.trial169.stdout:1077: writing trial16941936-31 from 621991 to 1420379 tid 1 2026-02-25T08:46:28.174 INFO:tasks.rados.rados.0.trial169.stdout:1077: writing trial16941936-31 from 1936439 to 2575948 tid 2 2026-02-25T08:46:28.175 INFO:tasks.rados.rados.0.trial169.stdout:1078: read oid 38 snap -1 2026-02-25T08:46:28.175 INFO:tasks.rados.rados.0.trial169.stdout:1078: expect (ObjNum 495 snap 131 seq_num 495) 2026-02-25T08:46:28.175 INFO:tasks.rados.rados.0.trial169.stdout:1079: write oid 22 current snap is 138 2026-02-25T08:46:28.175 INFO:tasks.rados.rados.0.trial169.stdout:1079: seq_num 514 ranges {638798=473637,1594921=783167} 2026-02-25T08:46:28.179 INFO:tasks.rados.rados.0.trial169.stdout:1079: writing trial16941936-22 from 638798 to 1112435 tid 1 2026-02-25T08:46:28.183 INFO:tasks.rados.rados.0.trial169.stdout:1079: writing trial16941936-22 from 1594921 to 2378088 tid 2 2026-02-25T08:46:28.184 INFO:tasks.rados.rados.0.trial169.stdout:1080: snap_remove snap 123 2026-02-25T08:46:28.816 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:28.895 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:28.892+0000 7f4bea98c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:28.897 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:28.895+0000 7f4bea98c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:28.900 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:28.940 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:28.941 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:29.165 INFO:tasks.rados.rados.0.trial169.stdout:1080: done (3 left) 2026-02-25T08:46:29.165 INFO:tasks.rados.rados.0.trial169.stdout:1081: write (excl) oid 30 current snap is 138 2026-02-25T08:46:29.165 INFO:tasks.rados.rados.0.trial169.stdout:1081: seq_num 515 ranges {655605=548886,1653403=643178} 2026-02-25T08:46:29.169 INFO:tasks.rados.rados.0.trial169.stdout:1081: writing trial16941936-30 from 655605 to 1204491 tid 1 2026-02-25T08:46:29.174 INFO:tasks.rados.rados.0.trial169.stdout:1081: writing trial16941936-30 from 1653403 to 2296581 tid 2 2026-02-25T08:46:29.174 INFO:tasks.rados.rados.0.trial169.stdout:1082: write (excl) oid 35 current snap is 138 2026-02-25T08:46:29.174 INFO:tasks.rados.rados.0.trial169.stdout:1082: seq_num 516 ranges {672412=624135,1995532=676880} 2026-02-25T08:46:29.179 INFO:tasks.rados.rados.0.trial169.stdout:1082: writing trial16941936-35 from 672412 to 1296547 tid 1 2026-02-25T08:46:29.184 INFO:tasks.rados.rados.0.trial169.stdout:1082: writing trial16941936-35 from 1995532 to 2672412 tid 2 2026-02-25T08:46:29.184 INFO:tasks.rados.rados.0.trial169.stdout:1083: write (excl) oid 14 current snap is 138 2026-02-25T08:46:29.184 INFO:tasks.rados.rados.0.trial169.stdout:1078: expect (ObjNum 495 snap 131 seq_num 495) 2026-02-25T08:46:29.196 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:29.215 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial169.stdout:1077: finishing write tid 1 to trial16941936-31 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial169.stdout:1077: finishing write tid 2 to trial16941936-31 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial169.stdout:1077: finishing write tid 3 to trial16941936-31 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial169.stdout:1077: finishing write tid 4 to trial16941936-31 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial169.stdout:1077: oid 31 updating version 0 to 476 2026-02-25T08:46:29.235 INFO:tasks.rados.rados.0.trial169.stdout:1077: oid 31 version 476 is already newer than 476 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial169.stdout:1077: oid 31 version 476 is already newer than 474 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 31 v 476 (ObjNum 513 snap 138 seq_num 513) dirty exists 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial169.stdout:1077: left oid 31 (ObjNum 513 snap 138 seq_num 513) 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial169.stdout:1079: finishing write tid 1 to trial16941936-22 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial169.stdout:1079: finishing write tid 2 to trial16941936-22 2026-02-25T08:46:29.236 INFO:tasks.rados.rados.0.trial169.stdout:1083: seq_num 517 ranges {689219=415737,1770367=646847} 2026-02-25T08:46:29.239 INFO:tasks.rados.rados.0.trial169.stdout:1083: writing trial16941936-14 from 689219 to 1104956 tid 1 2026-02-25T08:46:29.244 INFO:tasks.rados.rados.0.trial169.stdout:1083: writing trial16941936-14 from 1770367 to 2417214 tid 2 2026-02-25T08:46:29.244 INFO:tasks.rados.rados.0.trial169.stdout:1077: done (5 left) 2026-02-25T08:46:29.245 INFO:tasks.rados.rados.0.trial169.stdout:1078: done (4 left) 2026-02-25T08:46:29.245 INFO:tasks.rados.rados.0.trial169.stdout:1084: read oid 25 snap -1 2026-02-25T08:46:29.245 INFO:tasks.rados.rados.0.trial169.stdout:1084: expect deleted 2026-02-25T08:46:29.245 INFO:tasks.rados.rados.0.trial169.stdout:1085: delete oid 1 current snap is 138 2026-02-25T08:46:29.245 INFO:tasks.rados.rados.0.trial169.stdout:1079: finishing write tid 3 to trial16941936-22 2026-02-25T08:46:29.245 INFO:tasks.rados.rados.0.trial169.stdout:1079: finishing write tid 4 to trial16941936-22 2026-02-25T08:46:29.245 INFO:tasks.rados.rados.0.trial169.stdout:1079: oid 22 updating version 0 to 197 2026-02-25T08:46:29.245 INFO:tasks.rados.rados.0.trial169.stdout:1079: oid 22 version 197 is already newer than 197 2026-02-25T08:46:29.245 INFO:tasks.rados.rados.0.trial169.stdout:1079: oid 22 version 197 is already newer than 197 2026-02-25T08:46:29.245 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 22 v 197 (ObjNum 514 snap 138 seq_num 514) dirty exists 2026-02-25T08:46:29.245 INFO:tasks.rados.rados.0.trial169.stdout:1079: left oid 22 (ObjNum 514 snap 138 seq_num 514) 2026-02-25T08:46:29.245 INFO:tasks.rados.rados.0.trial169.stdout:1081: finishing write tid 1 to trial16941936-30 2026-02-25T08:46:29.246 INFO:tasks.rados.rados.0.trial169.stdout:1082: finishing write tid 1 to trial16941936-35 2026-02-25T08:46:29.246 INFO:tasks.rados.rados.0.trial169.stdout:1081: finishing write tid 2 to trial16941936-30 2026-02-25T08:46:29.246 INFO:tasks.rados.rados.0.trial169.stdout:1082: finishing write tid 2 to trial16941936-35 2026-02-25T08:46:29.249 INFO:tasks.rados.rados.0.trial169.stdout:1082: finishing write tid 3 to trial16941936-35 2026-02-25T08:46:29.250 INFO:tasks.rados.rados.0.trial169.stdout:1081: finishing write tid 3 to trial16941936-30 2026-02-25T08:46:29.252 INFO:tasks.rados.rados.0.trial169.stdout:1079: done (5 left) 2026-02-25T08:46:29.252 INFO:tasks.rados.rados.0.trial169.stdout:1084: done (4 left) 2026-02-25T08:46:29.252 INFO:tasks.rados.rados.0.trial169.stdout:1085: done (3 left) 2026-02-25T08:46:29.252 INFO:tasks.rados.rados.0.trial169.stdout:1086: read oid 31 snap 136 2026-02-25T08:46:29.252 INFO:tasks.rados.rados.0.trial169.stdout:1086: expect (ObjNum 507 snap 136 seq_num 507) 2026-02-25T08:46:29.252 INFO:tasks.rados.rados.0.trial169.stdout:1087: read oid 45 snap -1 2026-02-25T08:46:29.252 INFO:tasks.rados.rados.0.trial169.stdout:1087: expect (ObjNum 506 snap 136 seq_num 506) 2026-02-25T08:46:29.252 INFO:tasks.rados.rados.0.trial169.stdout:1088: read oid 32 snap -1 2026-02-25T08:46:29.253 INFO:tasks.rados.rados.0.trial169.stdout:1088: expect (ObjNum 478 snap 126 seq_num 478) 2026-02-25T08:46:29.253 INFO:tasks.rados.rados.0.trial169.stdout:1089: write oid 24 current snap is 138 2026-02-25T08:46:29.253 INFO:tasks.rados.rados.0.trial169.stdout:1089: seq_num 519 ranges {722833=566235,1887331=650516} 2026-02-25T08:46:29.256 INFO:tasks.rados.rados.0.trial169.stdout:1089: writing trial16941936-24 from 722833 to 1289068 tid 1 2026-02-25T08:46:29.261 INFO:tasks.rados.rados.0.trial169.stdout:1089: writing trial16941936-24 from 1887331 to 2537847 tid 2 2026-02-25T08:46:29.261 INFO:tasks.rados.rados.0.trial169.stdout:1090: write (excl) oid 18 current snap is 138 2026-02-25T08:46:29.261 INFO:tasks.rados.rados.0.trial169.stdout:1090: seq_num 520 ranges {739640=641484,1829460=794174} 2026-02-25T08:46:29.266 INFO:tasks.rados.rados.0.trial169.stdout:1090: writing trial16941936-18 from 739640 to 1381124 tid 1 2026-02-25T08:46:29.272 INFO:tasks.rados.rados.0.trial169.stdout:1090: writing trial16941936-18 from 1829460 to 2623634 tid 2 2026-02-25T08:46:29.272 INFO:tasks.rados.rados.0.trial169.stdout:1091: read oid 48 snap 131 2026-02-25T08:46:29.272 INFO:tasks.rados.rados.0.trial169.stdout:1091: expect deleted 2026-02-25T08:46:29.272 INFO:tasks.rados.rados.0.trial169.stdout:1092: snap_create 2026-02-25T08:46:29.272 INFO:tasks.rados.rados.0.trial169.stdout:1081: finishing write tid 4 to trial16941936-30 2026-02-25T08:46:29.272 INFO:tasks.rados.rados.0.trial169.stdout:1081: oid 30 updating version 0 to 158 2026-02-25T08:46:29.272 INFO:tasks.rados.rados.0.trial169.stdout:1081: oid 30 version 158 is already newer than 156 2026-02-25T08:46:29.272 INFO:tasks.rados.rados.0.trial169.stdout:1081: oid 30 version 158 is already newer than 158 2026-02-25T08:46:29.272 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 30 v 158 (ObjNum 515 snap 138 seq_num 515) dirty exists 2026-02-25T08:46:29.272 INFO:tasks.rados.rados.0.trial169.stdout:1081: left oid 30 (ObjNum 515 snap 138 seq_num 515) 2026-02-25T08:46:29.273 INFO:tasks.rados.rados.0.trial169.stdout:1082: finishing write tid 4 to trial16941936-35 2026-02-25T08:46:29.273 INFO:tasks.rados.rados.0.trial169.stdout:1082: oid 35 updating version 0 to 564 2026-02-25T08:46:29.273 INFO:tasks.rados.rados.0.trial169.stdout:1082: oid 35 updating version 564 to 566 2026-02-25T08:46:29.273 INFO:tasks.rados.rados.0.trial169.stdout:1082: oid 35 version 566 is already newer than 566 2026-02-25T08:46:29.273 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 35 v 566 (ObjNum 516 snap 138 seq_num 516) dirty exists 2026-02-25T08:46:29.273 INFO:tasks.rados.rados.0.trial169.stdout:1082: left oid 35 (ObjNum 516 snap 138 seq_num 516) 2026-02-25T08:46:29.273 INFO:tasks.rados.rados.0.trial169.stdout:1086: expect (ObjNum 507 snap 136 seq_num 507) 2026-02-25T08:46:29.273 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:29.352 INFO:tasks.rados.rados.0.trial169.stdout:1088: expect (ObjNum 478 snap 126 seq_num 478) 2026-02-25T08:46:29.354 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 557545 2026-02-25T08:46:29.367 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:29.386 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1649286 2026-02-25T08:46:29.395 INFO:tasks.rados.rados.0.trial169.stdout:1083: finishing write tid 1 to trial16941936-14 2026-02-25T08:46:29.395 INFO:tasks.rados.rados.0.trial169.stdout:1089: finishing write tid 1 to trial16941936-24 2026-02-25T08:46:29.395 INFO:tasks.rados.rados.0.trial169.stdout:1089: finishing write tid 2 to trial16941936-24 2026-02-25T08:46:29.395 INFO:tasks.rados.rados.0.trial169.stdout:1083: finishing write tid 2 to trial16941936-14 2026-02-25T08:46:29.395 INFO:tasks.rados.rados.0.trial169.stdout:1089: finishing write tid 3 to trial16941936-24 2026-02-25T08:46:29.395 INFO:tasks.rados.rados.0.trial169.stdout:1089: finishing write tid 4 to trial16941936-24 2026-02-25T08:46:29.395 INFO:tasks.rados.rados.0.trial169.stdout:1089: oid 24 updating version 0 to 88 2026-02-25T08:46:29.395 INFO:tasks.rados.rados.0.trial169.stdout:1089: oid 24 version 88 is already newer than 88 2026-02-25T08:46:29.395 INFO:tasks.rados.rados.0.trial169.stdout:1089: oid 24 version 88 is already newer than 88 2026-02-25T08:46:29.395 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 24 v 88 (ObjNum 519 snap 138 seq_num 519) dirty exists 2026-02-25T08:46:29.396 INFO:tasks.rados.rados.0.trial169.stdout:1089: left oid 24 (ObjNum 519 snap 138 seq_num 519) 2026-02-25T08:46:29.396 INFO:tasks.rados.rados.0.trial169.stdout:1083: finishing write tid 3 to trial16941936-14 2026-02-25T08:46:29.396 INFO:tasks.rados.rados.0.trial169.stdout:1083: finishing write tid 4 to trial16941936-14 2026-02-25T08:46:29.396 INFO:tasks.rados.rados.0.trial169.stdout:1083: oid 14 updating version 0 to 175 2026-02-25T08:46:29.396 INFO:tasks.rados.rados.0.trial169.stdout:1083: oid 14 updating version 175 to 177 2026-02-25T08:46:29.396 INFO:tasks.rados.rados.0.trial169.stdout:1083: oid 14 version 177 is already newer than 177 2026-02-25T08:46:29.396 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 14 v 177 (ObjNum 517 snap 138 seq_num 517) dirty exists 2026-02-25T08:46:29.396 INFO:tasks.rados.rados.0.trial169.stdout:1083: left oid 14 (ObjNum 517 snap 138 seq_num 517) 2026-02-25T08:46:29.396 INFO:tasks.rados.rados.0.trial169.stdout:1090: finishing write tid 1 to trial16941936-18 2026-02-25T08:46:29.396 INFO:tasks.rados.rados.0.trial169.stdout:1087: expect (ObjNum 506 snap 136 seq_num 506) 2026-02-25T08:46:29.398 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 536081 2026-02-25T08:46:29.496 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1848739 2026-02-25T08:46:29.516 INFO:tasks.rados.rados.0.trial169.stdout:1090: finishing write tid 2 to trial16941936-18 2026-02-25T08:46:29.516 INFO:tasks.rados.rados.0.trial169.stdout:1090: finishing write tid 3 to trial16941936-18 2026-02-25T08:46:29.516 INFO:tasks.rados.rados.0.trial169.stdout:1090: finishing write tid 4 to trial16941936-18 2026-02-25T08:46:29.516 INFO:tasks.rados.rados.0.trial169.stdout:1090: oid 18 updating version 0 to 234 2026-02-25T08:46:29.516 INFO:tasks.rados.rados.0.trial169.stdout:1090: oid 18 updating version 234 to 236 2026-02-25T08:46:29.516 INFO:tasks.rados.rados.0.trial169.stdout:1090: oid 18 version 236 is already newer than 236 2026-02-25T08:46:29.517 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 18 v 236 (ObjNum 520 snap 138 seq_num 520) dirty exists 2026-02-25T08:46:29.517 INFO:tasks.rados.rados.0.trial169.stdout:1090: left oid 18 (ObjNum 520 snap 138 seq_num 520) 2026-02-25T08:46:30.167 INFO:tasks.rados.rados.0.trial169.stdout:1081: done (9 left) 2026-02-25T08:46:30.167 INFO:tasks.rados.rados.0.trial169.stdout:1082: done (8 left) 2026-02-25T08:46:30.167 INFO:tasks.rados.rados.0.trial169.stdout:1083: done (7 left) 2026-02-25T08:46:30.167 INFO:tasks.rados.rados.0.trial169.stdout:1086: done (6 left) 2026-02-25T08:46:30.167 INFO:tasks.rados.rados.0.trial169.stdout:1087: done (5 left) 2026-02-25T08:46:30.167 INFO:tasks.rados.rados.0.trial169.stdout:1088: done (4 left) 2026-02-25T08:46:30.167 INFO:tasks.rados.rados.0.trial169.stdout:1089: done (3 left) 2026-02-25T08:46:30.168 INFO:tasks.rados.rados.0.trial169.stdout:1090: done (2 left) 2026-02-25T08:46:30.168 INFO:tasks.rados.rados.0.trial169.stdout:1091: done (1 left) 2026-02-25T08:46:30.168 INFO:tasks.rados.rados.0.trial169.stdout:1092: done (0 left) 2026-02-25T08:46:30.168 INFO:tasks.rados.rados.0.trial169.stdout:1093: write oid 34 current snap is 139 2026-02-25T08:46:30.168 INFO:tasks.rados.rados.0.trial169.stdout:1093: seq_num 521 ranges {756447=716733,1887942=537832} 2026-02-25T08:46:30.171 INFO:tasks.rados.rados.0.trial169.stdout:1093: writing trial16941936-34 from 756447 to 1473180 tid 1 2026-02-25T08:46:30.176 INFO:tasks.rados.rados.0.trial169.stdout:1093: writing trial16941936-34 from 1887942 to 2425774 tid 2 2026-02-25T08:46:30.176 INFO:tasks.rados.rados.0.trial169.stdout:1094: write (excl) oid 29 current snap is 139 2026-02-25T08:46:30.176 INFO:tasks.rados.rados.0.trial169.stdout:1094: seq_num 522 ranges {773254=791982,2346424=426830} 2026-02-25T08:46:30.181 INFO:tasks.rados.rados.0.trial169.stdout:1094: writing trial16941936-29 from 773254 to 1565236 tid 1 2026-02-25T08:46:30.186 INFO:tasks.rados.rados.0.trial169.stdout:1094: writing trial16941936-29 from 2346424 to 2773254 tid 2 2026-02-25T08:46:30.186 INFO:tasks.rados.rados.0.trial169.stdout:1095: snap_create 2026-02-25T08:46:30.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~264s 2026-02-25T08:46:30.203 INFO:tasks.rados.rados.0.trial169.stdout:1094: finishing write tid 1 to trial16941936-29 2026-02-25T08:46:30.217 INFO:tasks.rados.rados.0.trial169.stdout:1094: finishing write tid 2 to trial16941936-29 2026-02-25T08:46:30.225 INFO:tasks.rados.rados.0.trial169.stdout:1094: finishing write tid 3 to trial16941936-29 2026-02-25T08:46:30.227 INFO:tasks.rados.rados.0.trial169.stdout:1094: finishing write tid 4 to trial16941936-29 2026-02-25T08:46:30.227 INFO:tasks.rados.rados.0.trial169.stdout:1094: oid 29 updating version 0 to 569 2026-02-25T08:46:30.227 INFO:tasks.rados.rados.0.trial169.stdout:1094: oid 29 version 569 is already newer than 569 2026-02-25T08:46:30.227 INFO:tasks.rados.rados.0.trial169.stdout:1094: oid 29 version 569 is already newer than 567 2026-02-25T08:46:30.228 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 29 v 569 (ObjNum 522 snap 139 seq_num 522) dirty exists 2026-02-25T08:46:30.228 INFO:tasks.rados.rados.0.trial169.stdout:1094: left oid 29 (ObjNum 522 snap 139 seq_num 522) 2026-02-25T08:46:30.242 INFO:tasks.rados.rados.0.trial169.stdout:1093: finishing write tid 1 to trial16941936-34 2026-02-25T08:46:30.248 INFO:tasks.rados.rados.0.trial169.stdout:1093: finishing write tid 2 to trial16941936-34 2026-02-25T08:46:30.251 INFO:tasks.rados.rados.0.trial169.stdout:1093: finishing write tid 3 to trial16941936-34 2026-02-25T08:46:30.253 INFO:tasks.rados.rados.0.trial169.stdout:1093: finishing write tid 4 to trial16941936-34 2026-02-25T08:46:30.253 INFO:tasks.rados.rados.0.trial169.stdout:1093: oid 34 updating version 0 to 172 2026-02-25T08:46:30.253 INFO:tasks.rados.rados.0.trial169.stdout:1093: oid 34 version 172 is already newer than 172 2026-02-25T08:46:30.254 INFO:tasks.rados.rados.0.trial169.stdout:1093: oid 34 version 172 is already newer than 172 2026-02-25T08:46:30.254 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 34 v 172 (ObjNum 521 snap 139 seq_num 521) dirty exists 2026-02-25T08:46:30.254 INFO:tasks.rados.rados.0.trial169.stdout:1093: left oid 34 (ObjNum 521 snap 139 seq_num 521) 2026-02-25T08:46:30.943 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:31.020 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:31.017+0000 7f852001a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:31.023 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:31.021+0000 7f852001a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:31.026 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:31.068 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:31.068 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:31.165 INFO:tasks.rados.rados.0.trial169.stdout:1093: done (2 left) 2026-02-25T08:46:31.166 INFO:tasks.rados.rados.0.trial169.stdout:1094: done (1 left) 2026-02-25T08:46:31.166 INFO:tasks.rados.rados.0.trial169.stdout:1095: done (0 left) 2026-02-25T08:46:31.166 INFO:tasks.rados.rados.0.trial169.stdout:1096: write oid 14 current snap is 140 2026-02-25T08:46:31.166 INFO:tasks.rados.rados.0.trial169.stdout:1096: seq_num 523 ranges {790061=467231,2004906=541501} 2026-02-25T08:46:31.170 INFO:tasks.rados.rados.0.trial169.stdout:1096: writing trial16941936-14 from 790061 to 1257292 tid 1 2026-02-25T08:46:31.175 INFO:tasks.rados.rados.0.trial169.stdout:1096: writing trial16941936-14 from 2004906 to 2546407 tid 2 2026-02-25T08:46:31.175 INFO:tasks.rados.rados.0.trial169.stdout:1097: snap_create 2026-02-25T08:46:31.221 INFO:tasks.rados.rados.0.trial169.stdout:1096: finishing write tid 1 to trial16941936-14 2026-02-25T08:46:31.230 INFO:tasks.rados.rados.0.trial169.stdout:1096: finishing write tid 2 to trial16941936-14 2026-02-25T08:46:31.235 INFO:tasks.rados.rados.0.trial169.stdout:1096: finishing write tid 3 to trial16941936-14 2026-02-25T08:46:31.237 INFO:tasks.rados.rados.0.trial169.stdout:1096: finishing write tid 4 to trial16941936-14 2026-02-25T08:46:31.238 INFO:tasks.rados.rados.0.trial169.stdout:1096: oid 14 updating version 0 to 181 2026-02-25T08:46:31.238 INFO:tasks.rados.rados.0.trial169.stdout:1096: oid 14 version 181 is already newer than 181 2026-02-25T08:46:31.238 INFO:tasks.rados.rados.0.trial169.stdout:1096: oid 14 version 181 is already newer than 181 2026-02-25T08:46:31.238 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 14 v 181 (ObjNum 523 snap 140 seq_num 523) dirty exists 2026-02-25T08:46:31.238 INFO:tasks.rados.rados.0.trial169.stdout:1096: left oid 14 (ObjNum 523 snap 140 seq_num 523) 2026-02-25T08:46:31.390 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:46:31.462 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:31.465 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:46:32.168 INFO:tasks.rados.rados.0.trial169.stdout:1096: done (1 left) 2026-02-25T08:46:32.168 INFO:tasks.rados.rados.0.trial169.stdout:1097: done (0 left) 2026-02-25T08:46:32.169 INFO:tasks.rados.rados.0.trial169.stdout:1098: write oid 42 current snap is 141 2026-02-25T08:46:32.169 INFO:tasks.rados.rados.0.trial169.stdout:1098: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2026-02-25T08:46:32.172 INFO:tasks.rados.rados.0.trial169.stdout:1098: writing trial16941936-42 from 406868 to 949348 tid 1 2026-02-25T08:46:32.175 INFO:tasks.rados.rados.0.trial169.stdout:1098: writing trial16941936-42 from 1547035 to 1948547 tid 2 2026-02-25T08:46:32.179 INFO:tasks.rados.rados.0.trial169.stdout:1098: writing trial16941936-42 from 2690354 to 2806868 tid 3 2026-02-25T08:46:32.179 INFO:tasks.rados.rados.0.trial169.stdout:1099: snap_remove snap 133 2026-02-25T08:46:33.070 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:33.147 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:33.144+0000 7ff8c0a8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:33.149 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:33.148+0000 7ff8c0a8b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:33.153 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:33.167 INFO:tasks.rados.rados.0.trial169.stdout:1099: done (1 left) 2026-02-25T08:46:33.167 INFO:tasks.rados.rados.0.trial169.stdout:1100: read oid 39 snap -1 2026-02-25T08:46:33.167 INFO:tasks.rados.rados.0.trial169.stdout:1100: expect deleted 2026-02-25T08:46:33.167 INFO:tasks.rados.rados.0.trial169.stdout:1101: write oid 8 current snap is 141 2026-02-25T08:46:33.167 INFO:tasks.rados.rados.0.trial169.stdout:1098: finishing write tid 1 to trial16941936-42 2026-02-25T08:46:33.167 INFO:tasks.rados.rados.0.trial169.stdout:1101: seq_num 525 ranges {423675=734082,1721870=545170,2717777=105898} 2026-02-25T08:46:33.171 INFO:tasks.rados.rados.0.trial169.stdout:1101: writing trial16941936-8 from 423675 to 1157757 tid 1 2026-02-25T08:46:33.175 INFO:tasks.rados.rados.0.trial169.stdout:1101: writing trial16941936-8 from 1721870 to 2267040 tid 2 2026-02-25T08:46:33.178 INFO:tasks.rados.rados.0.trial169.stdout:1101: writing trial16941936-8 from 2717777 to 2823675 tid 3 2026-02-25T08:46:33.178 INFO:tasks.rados.rados.0.trial169.stdout:1102: snap_create 2026-02-25T08:46:33.178 INFO:tasks.rados.rados.0.trial169.stdout:1098: finishing write tid 2 to trial16941936-42 2026-02-25T08:46:33.178 INFO:tasks.rados.rados.0.trial169.stdout:1098: finishing write tid 3 to trial16941936-42 2026-02-25T08:46:33.178 INFO:tasks.rados.rados.0.trial169.stdout:1098: finishing write tid 4 to trial16941936-42 2026-02-25T08:46:33.178 INFO:tasks.rados.rados.0.trial169.stdout:1098: finishing write tid 5 to trial16941936-42 2026-02-25T08:46:33.178 INFO:tasks.rados.rados.0.trial169.stdout:1098: oid 42 updating version 0 to 186 2026-02-25T08:46:33.178 INFO:tasks.rados.rados.0.trial169.stdout:1098: oid 42 version 186 is already newer than 186 2026-02-25T08:46:33.178 INFO:tasks.rados.rados.0.trial169.stdout:1098: oid 42 version 186 is already newer than 186 2026-02-25T08:46:33.178 INFO:tasks.rados.rados.0.trial169.stdout:1098: oid 42 version 186 is already newer than 186 2026-02-25T08:46:33.178 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 42 v 186 (ObjNum 524 snap 141 seq_num 524) dirty exists 2026-02-25T08:46:33.178 INFO:tasks.rados.rados.0.trial169.stdout:1098: left oid 42 (ObjNum 524 snap 141 seq_num 524) 2026-02-25T08:46:33.247 INFO:tasks.rados.rados.0.trial169.stdout:1101: finishing write tid 1 to trial16941936-8 2026-02-25T08:46:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:33.252 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:33.256 INFO:tasks.rados.rados.0.trial169.stdout:1101: finishing write tid 2 to trial16941936-8 2026-02-25T08:46:33.264 INFO:tasks.rados.rados.0.trial169.stdout:1101: finishing write tid 3 to trial16941936-8 2026-02-25T08:46:33.267 INFO:tasks.rados.rados.0.trial169.stdout:1101: finishing write tid 4 to trial16941936-8 2026-02-25T08:46:33.268 INFO:tasks.rados.rados.0.trial169.stdout:1101: finishing write tid 5 to trial16941936-8 2026-02-25T08:46:33.268 INFO:tasks.rados.rados.0.trial169.stdout:1101: oid 8 updating version 0 to 488 2026-02-25T08:46:33.268 INFO:tasks.rados.rados.0.trial169.stdout:1101: oid 8 version 488 is already newer than 488 2026-02-25T08:46:33.268 INFO:tasks.rados.rados.0.trial169.stdout:1101: oid 8 version 488 is already newer than 488 2026-02-25T08:46:33.268 INFO:tasks.rados.rados.0.trial169.stdout:1101: oid 8 version 488 is already newer than 488 2026-02-25T08:46:33.269 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 8 v 488 (ObjNum 525 snap 141 seq_num 525) dirty exists 2026-02-25T08:46:33.269 INFO:tasks.rados.rados.0.trial169.stdout:1101: left oid 8 (ObjNum 525 snap 141 seq_num 525) 2026-02-25T08:46:34.169 INFO:tasks.rados.rados.0.trial169.stdout:1098: done (3 left) 2026-02-25T08:46:34.169 INFO:tasks.rados.rados.0.trial169.stdout:1100: done (2 left) 2026-02-25T08:46:34.169 INFO:tasks.rados.rados.0.trial169.stdout:1101: done (1 left) 2026-02-25T08:46:34.169 INFO:tasks.rados.rados.0.trial169.stdout:1102: done (0 left) 2026-02-25T08:46:34.169 INFO:tasks.rados.rados.0.trial169.stdout:1103: snap_remove snap 137 2026-02-25T08:46:35.168 INFO:tasks.rados.rados.0.trial169.stdout:1103: done (0 left) 2026-02-25T08:46:35.168 INFO:tasks.rados.rados.0.trial169.stdout:1104: delete oid 13 current snap is 142 2026-02-25T08:46:35.193 INFO:tasks.rados.rados.0.trial169.stdout:1104: done (0 left) 2026-02-25T08:46:35.193 INFO:tasks.rados.rados.0.trial169.stdout:1105: write (excl) oid 10 current snap is 142 2026-02-25T08:46:35.194 INFO:tasks.rados.rados.0.trial169.stdout:1105: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2026-02-25T08:46:35.197 INFO:tasks.rados.rados.0.trial169.stdout:1105: writing trial16941936-10 from 457289 to 941869 tid 1 2026-02-25T08:46:35.201 INFO:tasks.rados.rados.0.trial169.stdout:1105: writing trial16941936-10 from 1438834 to 1987673 tid 2 2026-02-25T08:46:35.204 INFO:tasks.rados.rados.0.trial169.stdout:1105: writing trial16941936-10 from 2772623 to 2857289 tid 3 2026-02-25T08:46:35.204 INFO:tasks.rados.rados.0.trial169.stdout:1106: write (excl) oid 1 current snap is 142 2026-02-25T08:46:35.204 INFO:tasks.rados.rados.0.trial169.stdout:1106: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2026-02-25T08:46:35.208 INFO:tasks.rados.rados.0.trial169.stdout:1106: writing trial16941936-1 from 474096 to 1033925 tid 1 2026-02-25T08:46:35.212 INFO:tasks.rados.rados.0.trial169.stdout:1106: writing trial16941936-1 from 1780963 to 2189813 tid 2 2026-02-25T08:46:35.215 INFO:tasks.rados.rados.0.trial169.stdout:1106: writing trial16941936-1 from 2800046 to 2874096 tid 3 2026-02-25T08:46:35.215 INFO:tasks.rados.rados.0.trial169.stdout:1107: read oid 29 snap -1 2026-02-25T08:46:35.215 INFO:tasks.rados.rados.0.trial169.stdout:1107: expect (ObjNum 522 snap 139 seq_num 522) 2026-02-25T08:46:35.216 INFO:tasks.rados.rados.0.trial169.stdout:1108: read oid 44 snap -1 2026-02-25T08:46:35.216 INFO:tasks.rados.rados.0.trial169.stdout:1108: expect deleted 2026-02-25T08:46:35.216 INFO:tasks.rados.rados.0.trial169.stdout:1109: snap_remove snap 139 2026-02-25T08:46:35.253 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:35.329 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:35.326+0000 7f82e5bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:35.331 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:35.329+0000 7f82e5bb8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:35.335 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:35.376 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:35.376 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:35.703 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~270s 2026-02-25T08:46:36.173 INFO:tasks.rados.rados.0.trial169.stdout:1109: done (4 left) 2026-02-25T08:46:36.173 INFO:tasks.rados.rados.0.trial169.stdout:1110: write (excl) oid 14 current snap is 142 2026-02-25T08:46:36.173 INFO:tasks.rados.rados.0.trial169.stdout:1105: finishing write tid 1 to trial16941936-10 2026-02-25T08:46:36.173 INFO:tasks.rados.rados.0.trial169.stdout:1110: seq_num 529 ranges {490903=635078,1839445=552508} 2026-02-25T08:46:36.177 INFO:tasks.rados.rados.0.trial169.stdout:1110: writing trial16941936-14 from 490903 to 1125981 tid 1 2026-02-25T08:46:36.184 INFO:tasks.rados.rados.0.trial169.stdout:1110: writing trial16941936-14 from 1839445 to 2391953 tid 2 2026-02-25T08:46:36.184 INFO:tasks.rados.rados.0.trial169.stdout:1106: finishing write tid 1 to trial16941936-1 2026-02-25T08:46:36.184 INFO:tasks.rados.rados.0.trial169.stdout:1105: finishing write tid 2 to trial16941936-10 2026-02-25T08:46:36.184 INFO:tasks.rados.rados.0.trial169.stdout:1105: finishing write tid 3 to trial16941936-10 2026-02-25T08:46:36.184 INFO:tasks.rados.rados.0.trial169.stdout:1111: read oid 12 snap -1 2026-02-25T08:46:36.184 INFO:tasks.rados.rados.0.trial169.stdout:1111: expect (ObjNum 346 snap 95 seq_num 346) 2026-02-25T08:46:36.184 INFO:tasks.rados.rados.0.trial169.stdout:1108: done (5 left) 2026-02-25T08:46:36.184 INFO:tasks.rados.rados.0.trial169.stdout:1112: delete oid 33 current snap is 142 2026-02-25T08:46:36.184 INFO:tasks.rados.rados.0.trial169.stdout:1105: finishing write tid 4 to trial16941936-10 2026-02-25T08:46:36.185 INFO:tasks.rados.rados.0.trial169.stdout:1105: finishing write tid 5 to trial16941936-10 2026-02-25T08:46:36.185 INFO:tasks.rados.rados.0.trial169.stdout:1105: oid 10 updating version 0 to 87 2026-02-25T08:46:36.185 INFO:tasks.rados.rados.0.trial169.stdout:1105: oid 10 version 87 is already newer than 87 2026-02-25T08:46:36.185 INFO:tasks.rados.rados.0.trial169.stdout:1105: oid 10 version 87 is already newer than 84 2026-02-25T08:46:36.185 INFO:tasks.rados.rados.0.trial169.stdout:1105: oid 10 version 87 is already newer than 87 2026-02-25T08:46:36.185 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 10 v 87 (ObjNum 527 snap 142 seq_num 527) dirty exists 2026-02-25T08:46:36.185 INFO:tasks.rados.rados.0.trial169.stdout:1105: left oid 10 (ObjNum 527 snap 142 seq_num 527) 2026-02-25T08:46:36.185 INFO:tasks.rados.rados.0.trial169.stdout:1106: finishing write tid 2 to trial16941936-1 2026-02-25T08:46:36.185 INFO:tasks.rados.rados.0.trial169.stdout:1107: expect (ObjNum 522 snap 139 seq_num 522) 2026-02-25T08:46:36.194 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:36.217 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:36.241 INFO:tasks.rados.rados.0.trial169.stdout:1106: finishing write tid 3 to trial16941936-1 2026-02-25T08:46:36.242 INFO:tasks.rados.rados.0.trial169.stdout:1106: finishing write tid 4 to trial16941936-1 2026-02-25T08:46:36.242 INFO:tasks.rados.rados.0.trial169.stdout:1106: finishing write tid 5 to trial16941936-1 2026-02-25T08:46:36.242 INFO:tasks.rados.rados.0.trial169.stdout:1106: oid 1 updating version 0 to 111 2026-02-25T08:46:36.242 INFO:tasks.rados.rados.0.trial169.stdout:1106: oid 1 updating version 111 to 114 2026-02-25T08:46:36.242 INFO:tasks.rados.rados.0.trial169.stdout:1106: oid 1 version 114 is already newer than 114 2026-02-25T08:46:36.242 INFO:tasks.rados.rados.0.trial169.stdout:1106: oid 1 version 114 is already newer than 114 2026-02-25T08:46:36.242 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 1 v 114 (ObjNum 528 snap 142 seq_num 528) dirty exists 2026-02-25T08:46:36.242 INFO:tasks.rados.rados.0.trial169.stdout:1106: left oid 1 (ObjNum 528 snap 142 seq_num 528) 2026-02-25T08:46:36.242 INFO:tasks.rados.rados.0.trial169.stdout:1105: done (5 left) 2026-02-25T08:46:36.242 INFO:tasks.rados.rados.0.trial169.stdout:1106: done (4 left) 2026-02-25T08:46:36.242 INFO:tasks.rados.rados.0.trial169.stdout:1107: done (3 left) 2026-02-25T08:46:36.243 INFO:tasks.rados.rados.0.trial169.stdout:1112: done (2 left) 2026-02-25T08:46:36.243 INFO:tasks.rados.rados.0.trial169.stdout:1113: write oid 49 current snap is 142 2026-02-25T08:46:36.243 INFO:tasks.rados.rados.0.trial169.stdout:1111: expect (ObjNum 346 snap 95 seq_num 346) 2026-02-25T08:46:36.264 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2471882 2026-02-25T08:46:36.362 INFO:tasks.rados.rados.0.trial169.stdout:1110: finishing write tid 1 to trial16941936-14 2026-02-25T08:46:36.362 INFO:tasks.rados.rados.0.trial169.stdout:1110: finishing write tid 2 to trial16941936-14 2026-02-25T08:46:36.362 INFO:tasks.rados.rados.0.trial169.stdout:1110: finishing write tid 3 to trial16941936-14 2026-02-25T08:46:36.363 INFO:tasks.rados.rados.0.trial169.stdout:1110: finishing write tid 4 to trial16941936-14 2026-02-25T08:46:36.363 INFO:tasks.rados.rados.0.trial169.stdout:1110: oid 14 updating version 0 to 188 2026-02-25T08:46:36.363 INFO:tasks.rados.rados.0.trial169.stdout:1110: oid 14 updating version 188 to 190 2026-02-25T08:46:36.363 INFO:tasks.rados.rados.0.trial169.stdout:1110: oid 14 version 190 is already newer than 190 2026-02-25T08:46:36.363 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 14 v 190 (ObjNum 529 snap 142 seq_num 529) dirty exists 2026-02-25T08:46:36.363 INFO:tasks.rados.rados.0.trial169.stdout:1110: left oid 14 (ObjNum 529 snap 142 seq_num 529) 2026-02-25T08:46:36.363 INFO:tasks.rados.rados.0.trial169.stdout:1113: seq_num 531 ranges {524517=785576,1956409=556177} 2026-02-25T08:46:36.367 INFO:tasks.rados.rados.0.trial169.stdout:1113: writing trial16941936-49 from 524517 to 1310093 tid 1 2026-02-25T08:46:36.372 INFO:tasks.rados.rados.0.trial169.stdout:1113: writing trial16941936-49 from 1956409 to 2512586 tid 2 2026-02-25T08:46:36.372 INFO:tasks.rados.rados.0.trial169.stdout:1110: done (2 left) 2026-02-25T08:46:36.372 INFO:tasks.rados.rados.0.trial169.stdout:1111: done (1 left) 2026-02-25T08:46:36.372 INFO:tasks.rados.rados.0.trial169.stdout:1114: read oid 20 snap -1 2026-02-25T08:46:36.372 INFO:tasks.rados.rados.0.trial169.stdout:1114: expect deleted 2026-02-25T08:46:36.372 INFO:tasks.rados.rados.0.trial169.stdout:1115: delete oid 27 current snap is 142 2026-02-25T08:46:36.405 INFO:tasks.rados.rados.0.trial169.stdout:1113: finishing write tid 1 to trial16941936-49 2026-02-25T08:46:36.414 INFO:tasks.rados.rados.0.trial169.stdout:1113: finishing write tid 2 to trial16941936-49 2026-02-25T08:46:36.419 INFO:tasks.rados.rados.0.trial169.stdout:1113: finishing write tid 3 to trial16941936-49 2026-02-25T08:46:36.421 INFO:tasks.rados.rados.0.trial169.stdout:1113: finishing write tid 4 to trial16941936-49 2026-02-25T08:46:36.421 INFO:tasks.rados.rados.0.trial169.stdout:1113: oid 49 updating version 0 to 495 2026-02-25T08:46:36.422 INFO:tasks.rados.rados.0.trial169.stdout:1113: oid 49 version 495 is already newer than 495 2026-02-25T08:46:36.422 INFO:tasks.rados.rados.0.trial169.stdout:1113: oid 49 version 495 is already newer than 495 2026-02-25T08:46:36.422 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 49 v 495 (ObjNum 531 snap 142 seq_num 531) dirty exists 2026-02-25T08:46:36.422 INFO:tasks.rados.rados.0.trial169.stdout:1113: left oid 49 (ObjNum 531 snap 142 seq_num 531) 2026-02-25T08:46:36.433 INFO:tasks.rados.rados.0.trial169.stdout:1113: done (2 left) 2026-02-25T08:46:36.434 INFO:tasks.rados.rados.0.trial169.stdout:1114: done (1 left) 2026-02-25T08:46:36.434 INFO:tasks.rados.rados.0.trial169.stdout:1115: done (0 left) 2026-02-25T08:46:36.434 INFO:tasks.rados.rados.0.trial169.stdout:1116: read oid 36 snap -1 2026-02-25T08:46:36.434 INFO:tasks.rados.rados.0.trial169.stdout:1116: expect (ObjNum 450 snap 121 seq_num 450) 2026-02-25T08:46:36.434 INFO:tasks.rados.rados.0.trial169.stdout:1117: write (excl) oid 16 current snap is 142 2026-02-25T08:46:36.434 INFO:tasks.rados.rados.0.trial169.stdout:1117: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2026-02-25T08:46:36.437 INFO:tasks.rados.rados.0.trial169.stdout:1117: writing trial16941936-16 from 558131 to 1210558 tid 1 2026-02-25T08:46:36.441 INFO:tasks.rados.rados.0.trial169.stdout:1117: writing trial16941936-16 from 1673373 to 2233219 tid 2 2026-02-25T08:46:36.444 INFO:tasks.rados.rados.0.trial169.stdout:1117: writing trial16941936-16 from 2820808 to 2958131 tid 3 2026-02-25T08:46:36.444 INFO:tasks.rados.rados.0.trial169.stdout:1118: snap_remove snap 140 2026-02-25T08:46:36.467 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:46:36.538 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:36.541 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:36.542 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:46:37.177 INFO:tasks.rados.rados.0.trial169.stdout:1118: done (2 left) 2026-02-25T08:46:37.177 INFO:tasks.rados.rados.0.trial169.stdout:1119: read oid 29 snap -1 2026-02-25T08:46:37.177 INFO:tasks.rados.rados.0.trial169.stdout:1119: expect (ObjNum 522 snap 139 seq_num 522) 2026-02-25T08:46:37.178 INFO:tasks.rados.rados.0.trial169.stdout:1120: snap_remove snap 134 2026-02-25T08:46:37.378 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:37.456 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:37.453+0000 7f926d37e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:37.458 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:37.457+0000 7f926d37e640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:37.462 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:37.501 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:38.166 INFO:tasks.rados.rados.0.trial169.stdout:1120: done (3 left) 2026-02-25T08:46:38.166 INFO:tasks.rados.rados.0.trial169.stdout:1121: delete oid 18 current snap is 142 2026-02-25T08:46:38.166 INFO:tasks.rados.rados.0.trial169.stdout:1116: expect (ObjNum 450 snap 121 seq_num 450) 2026-02-25T08:46:38.167 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:38.291 INFO:tasks.rados.rados.0.trial169.stdout:1117: finishing write tid 1 to trial16941936-16 2026-02-25T08:46:38.291 INFO:tasks.rados.rados.0.trial169.stdout:1117: finishing write tid 2 to trial16941936-16 2026-02-25T08:46:38.291 INFO:tasks.rados.rados.0.trial169.stdout:1117: finishing write tid 3 to trial16941936-16 2026-02-25T08:46:38.291 INFO:tasks.rados.rados.0.trial169.stdout:1116: done (3 left) 2026-02-25T08:46:38.291 INFO:tasks.rados.rados.0.trial169.stdout:1121: done (2 left) 2026-02-25T08:46:38.291 INFO:tasks.rados.rados.0.trial169.stdout:1122: write oid 7 current snap is 142 2026-02-25T08:46:38.291 INFO:tasks.rados.rados.0.trial169.stdout:1117: finishing write tid 4 to trial16941936-16 2026-02-25T08:46:38.291 INFO:tasks.rados.rados.0.trial169.stdout:1117: finishing write tid 5 to trial16941936-16 2026-02-25T08:46:38.292 INFO:tasks.rados.rados.0.trial169.stdout:1117: oid 16 updating version 0 to 212 2026-02-25T08:46:38.292 INFO:tasks.rados.rados.0.trial169.stdout:1117: oid 16 version 212 is already newer than 209 2026-02-25T08:46:38.292 INFO:tasks.rados.rados.0.trial169.stdout:1117: oid 16 version 212 is already newer than 212 2026-02-25T08:46:38.292 INFO:tasks.rados.rados.0.trial169.stdout:1117: oid 16 version 212 is already newer than 212 2026-02-25T08:46:38.292 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 16 v 212 (ObjNum 533 snap 142 seq_num 533) dirty exists 2026-02-25T08:46:38.292 INFO:tasks.rados.rados.0.trial169.stdout:1117: left oid 16 (ObjNum 533 snap 142 seq_num 533) 2026-02-25T08:46:38.292 INFO:tasks.rados.rados.0.trial169.stdout:1122: seq_num 535 ranges {591745=402925,1790337=563515} 2026-02-25T08:46:38.294 INFO:tasks.rados.rados.0.trial169.stdout:1122: writing trial16941936-7 from 591745 to 994670 tid 1 2026-02-25T08:46:38.299 INFO:tasks.rados.rados.0.trial169.stdout:1122: writing trial16941936-7 from 1790337 to 2353852 tid 2 2026-02-25T08:46:38.299 INFO:tasks.rados.rados.0.trial169.stdout:1117: done (2 left) 2026-02-25T08:46:38.300 INFO:tasks.rados.rados.0.trial169.stdout:1123: write oid 18 current snap is 142 2026-02-25T08:46:38.300 INFO:tasks.rados.rados.0.trial169.stdout:1123: seq_num 536 ranges {608552=478174,1848819=707173} 2026-02-25T08:46:38.303 INFO:tasks.rados.rados.0.trial169.stdout:1123: writing trial16941936-18 from 608552 to 1086726 tid 1 2026-02-25T08:46:38.309 INFO:tasks.rados.rados.0.trial169.stdout:1123: writing trial16941936-18 from 1848819 to 2555992 tid 2 2026-02-25T08:46:38.309 INFO:tasks.rados.rados.0.trial169.stdout:1124: delete oid 45 current snap is 142 2026-02-25T08:46:38.309 INFO:tasks.rados.rados.0.trial169.stdout:1119: expect (ObjNum 522 snap 139 seq_num 522) 2026-02-25T08:46:38.337 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1308294 2026-02-25T08:46:38.350 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 320977 2026-02-25T08:46:38.372 INFO:tasks.rados.rados.0.trial169.stdout:1119: done (3 left) 2026-02-25T08:46:38.372 INFO:tasks.rados.rados.0.trial169.stdout:1124: done (2 left) 2026-02-25T08:46:38.372 INFO:tasks.rados.rados.0.trial169.stdout:1125: delete oid 16 current snap is 142 2026-02-25T08:46:38.373 INFO:tasks.rados.rados.0.trial169.stdout:1122: finishing write tid 1 to trial16941936-7 2026-02-25T08:46:38.373 INFO:tasks.rados.rados.0.trial169.stdout:1122: finishing write tid 2 to trial16941936-7 2026-02-25T08:46:38.373 INFO:tasks.rados.rados.0.trial169.stdout:1122: finishing write tid 3 to trial16941936-7 2026-02-25T08:46:38.373 INFO:tasks.rados.rados.0.trial169.stdout:1122: finishing write tid 4 to trial16941936-7 2026-02-25T08:46:38.373 INFO:tasks.rados.rados.0.trial169.stdout:1122: oid 7 updating version 0 to 582 2026-02-25T08:46:38.373 INFO:tasks.rados.rados.0.trial169.stdout:1122: oid 7 version 582 is already newer than 582 2026-02-25T08:46:38.373 INFO:tasks.rados.rados.0.trial169.stdout:1122: oid 7 version 582 is already newer than 582 2026-02-25T08:46:38.373 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 7 v 582 (ObjNum 535 snap 142 seq_num 535) dirty exists 2026-02-25T08:46:38.374 INFO:tasks.rados.rados.0.trial169.stdout:1122: left oid 7 (ObjNum 535 snap 142 seq_num 535) 2026-02-25T08:46:38.374 INFO:tasks.rados.rados.0.trial169.stdout:1123: finishing write tid 1 to trial16941936-18 2026-02-25T08:46:38.374 INFO:tasks.rados.rados.0.trial169.stdout:1123: finishing write tid 2 to trial16941936-18 2026-02-25T08:46:38.374 INFO:tasks.rados.rados.0.trial169.stdout:1123: finishing write tid 3 to trial16941936-18 2026-02-25T08:46:38.374 INFO:tasks.rados.rados.0.trial169.stdout:1123: finishing write tid 4 to trial16941936-18 2026-02-25T08:46:38.374 INFO:tasks.rados.rados.0.trial169.stdout:1123: oid 18 updating version 0 to 255 2026-02-25T08:46:38.374 INFO:tasks.rados.rados.0.trial169.stdout:1123: oid 18 version 255 is already newer than 255 2026-02-25T08:46:38.374 INFO:tasks.rados.rados.0.trial169.stdout:1123: oid 18 version 255 is already newer than 255 2026-02-25T08:46:38.374 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 18 v 255 (ObjNum 536 snap 142 seq_num 536) dirty exists 2026-02-25T08:46:38.374 INFO:tasks.rados.rados.0.trial169.stdout:1123: left oid 18 (ObjNum 536 snap 142 seq_num 536) 2026-02-25T08:46:38.386 INFO:tasks.rados.rados.0.trial169.stdout:1122: done (2 left) 2026-02-25T08:46:38.386 INFO:tasks.rados.rados.0.trial169.stdout:1123: done (1 left) 2026-02-25T08:46:38.386 INFO:tasks.rados.rados.0.trial169.stdout:1125: done (0 left) 2026-02-25T08:46:38.386 INFO:tasks.rados.rados.0.trial169.stdout:1126: delete oid 30 current snap is 142 2026-02-25T08:46:38.413 INFO:tasks.rados.rados.0.trial169.stdout:1126: done (0 left) 2026-02-25T08:46:38.413 INFO:tasks.rados.rados.0.trial169.stdout:1127: read oid 36 snap -1 2026-02-25T08:46:38.414 INFO:tasks.rados.rados.0.trial169.stdout:1127: expect (ObjNum 450 snap 121 seq_num 450) 2026-02-25T08:46:38.414 INFO:tasks.rados.rados.0.trial169.stdout:1128: write (excl) oid 4 current snap is 142 2026-02-25T08:46:38.414 INFO:tasks.rados.rados.0.trial169.stdout:1128: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2026-02-25T08:46:38.418 INFO:tasks.rados.rados.0.trial169.stdout:1128: writing trial16941936-4 from 675780 to 1171303 tid 1 2026-02-25T08:46:38.423 INFO:tasks.rados.rados.0.trial169.stdout:1128: writing trial16941936-4 from 1682747 to 2397258 tid 2 2026-02-25T08:46:38.425 INFO:tasks.rados.rados.0.trial169.stdout:1128: writing trial16941936-4 from 3012769 to 3075780 tid 3 2026-02-25T08:46:38.425 INFO:tasks.rados.rados.0.trial169.stdout:1129: snap_create 2026-02-25T08:46:38.437 INFO:tasks.rados.rados.0.trial169.stdout:1128: finishing write tid 1 to trial16941936-4 2026-02-25T08:46:38.440 INFO:tasks.rados.rados.0.trial169.stdout:1127: expect (ObjNum 450 snap 121 seq_num 450) 2026-02-25T08:46:38.440 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:38.537 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2370084 2026-02-25T08:46:38.553 INFO:tasks.rados.rados.0.trial169.stdout:1128: finishing write tid 2 to trial16941936-4 2026-02-25T08:46:38.553 INFO:tasks.rados.rados.0.trial169.stdout:1128: finishing write tid 3 to trial16941936-4 2026-02-25T08:46:38.553 INFO:tasks.rados.rados.0.trial169.stdout:1128: finishing write tid 4 to trial16941936-4 2026-02-25T08:46:38.553 INFO:tasks.rados.rados.0.trial169.stdout:1128: finishing write tid 5 to trial16941936-4 2026-02-25T08:46:38.553 INFO:tasks.rados.rados.0.trial169.stdout:1128: oid 4 updating version 0 to 586 2026-02-25T08:46:38.553 INFO:tasks.rados.rados.0.trial169.stdout:1128: oid 4 version 586 is already newer than 586 2026-02-25T08:46:38.553 INFO:tasks.rados.rados.0.trial169.stdout:1128: oid 4 version 586 is already newer than 583 2026-02-25T08:46:38.554 INFO:tasks.rados.rados.0.trial169.stdout:1128: oid 4 version 586 is already newer than 586 2026-02-25T08:46:38.554 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 4 v 586 (ObjNum 540 snap 142 seq_num 540) dirty exists 2026-02-25T08:46:38.554 INFO:tasks.rados.rados.0.trial169.stdout:1128: left oid 4 (ObjNum 540 snap 142 seq_num 540) 2026-02-25T08:46:39.182 INFO:tasks.rados.rados.0.trial169.stdout:1127: done (2 left) 2026-02-25T08:46:39.182 INFO:tasks.rados.rados.0.trial169.stdout:1128: done (1 left) 2026-02-25T08:46:39.182 INFO:tasks.rados.rados.0.trial169.stdout:1129: done (0 left) 2026-02-25T08:46:39.182 INFO:tasks.rados.rados.0.trial169.stdout:1130: write (excl) oid 18 current snap is 143 2026-02-25T08:46:39.182 INFO:tasks.rados.rados.0.trial169.stdout:1130: seq_num 541 ranges {692587=570772,2024876=574522,3040192=52395} 2026-02-25T08:46:39.186 INFO:tasks.rados.rados.0.trial169.stdout:1130: writing trial16941936-18 from 692587 to 1263359 tid 1 2026-02-25T08:46:39.191 INFO:tasks.rados.rados.0.trial169.stdout:1130: writing trial16941936-18 from 2024876 to 2599398 tid 2 2026-02-25T08:46:39.193 INFO:tasks.rados.rados.0.trial169.stdout:1130: writing trial16941936-18 from 3040192 to 3092587 tid 3 2026-02-25T08:46:39.193 INFO:tasks.rados.rados.0.trial169.stdout:1131: snap_create 2026-02-25T08:46:39.226 INFO:tasks.rados.rados.0.trial169.stdout:1130: finishing write tid 1 to trial16941936-18 2026-02-25T08:46:39.239 INFO:tasks.rados.rados.0.trial169.stdout:1130: finishing write tid 2 to trial16941936-18 2026-02-25T08:46:39.247 INFO:tasks.rados.rados.0.trial169.stdout:1130: finishing write tid 3 to trial16941936-18 2026-02-25T08:46:39.247 INFO:tasks.rados.rados.0.trial169.stdout:1130: finishing write tid 4 to trial16941936-18 2026-02-25T08:46:39.249 INFO:tasks.rados.rados.0.trial169.stdout:1130: finishing write tid 5 to trial16941936-18 2026-02-25T08:46:39.249 INFO:tasks.rados.rados.0.trial169.stdout:1130: oid 18 updating version 0 to 257 2026-02-25T08:46:39.249 INFO:tasks.rados.rados.0.trial169.stdout:1130: oid 18 updating version 257 to 260 2026-02-25T08:46:39.249 INFO:tasks.rados.rados.0.trial169.stdout:1130: oid 18 version 260 is already newer than 260 2026-02-25T08:46:39.249 INFO:tasks.rados.rados.0.trial169.stdout:1130: oid 18 version 260 is already newer than 260 2026-02-25T08:46:39.249 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 18 v 260 (ObjNum 541 snap 143 seq_num 541) dirty exists 2026-02-25T08:46:39.250 INFO:tasks.rados.rados.0.trial169.stdout:1130: left oid 18 (ObjNum 541 snap 143 seq_num 541) 2026-02-25T08:46:39.502 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:39.580 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:39.577+0000 7f9c75774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:39.582 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:39.581+0000 7f9c75774640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:39.586 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:39.624 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:39.625 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:40.184 INFO:tasks.rados.rados.0.trial169.stdout:1130: done (1 left) 2026-02-25T08:46:40.184 INFO:tasks.rados.rados.0.trial169.stdout:1131: done (0 left) 2026-02-25T08:46:40.184 INFO:tasks.rados.rados.0.trial169.stdout:1132: snap_remove snap 142 2026-02-25T08:46:41.186 INFO:tasks.rados.rados.0.trial169.stdout:1132: done (0 left) 2026-02-25T08:46:41.187 INFO:tasks.rados.rados.0.trial169.stdout:1133: delete oid 28 current snap is 144 2026-02-25T08:46:41.207 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~275s 2026-02-25T08:46:41.234 INFO:tasks.rados.rados.0.trial169.stdout:1133: done (0 left) 2026-02-25T08:46:41.234 INFO:tasks.rados.rados.0.trial169.stdout:1134: write oid 17 current snap is 144 2026-02-25T08:46:41.235 INFO:tasks.rados.rados.0.trial169.stdout:1134: seq_num 543 ranges {726201=721270,2141840=578191} 2026-02-25T08:46:41.240 INFO:tasks.rados.rados.0.trial169.stdout:1134: writing trial16941936-17 from 726201 to 1447471 tid 1 2026-02-25T08:46:41.245 INFO:tasks.rados.rados.0.trial169.stdout:1134: writing trial16941936-17 from 2141840 to 2720031 tid 2 2026-02-25T08:46:41.245 INFO:tasks.rados.rados.0.trial169.stdout:1135: write (excl) oid 9 current snap is 144 2026-02-25T08:46:41.245 INFO:tasks.rados.rados.0.trial169.stdout:1135: seq_num 544 ranges {743008=796519,2200322=721849} 2026-02-25T08:46:41.251 INFO:tasks.rados.rados.0.trial169.stdout:1135: writing trial16941936-9 from 743008 to 1539527 tid 1 2026-02-25T08:46:41.256 INFO:tasks.rados.rados.0.trial169.stdout:1135: writing trial16941936-9 from 2200322 to 2922171 tid 2 2026-02-25T08:46:41.256 INFO:tasks.rados.rados.0.trial169.stdout:1136: read oid 6 snap -1 2026-02-25T08:46:41.257 INFO:tasks.rados.rados.0.trial169.stdout:1136: expect (ObjNum 499 snap 133 seq_num 499) 2026-02-25T08:46:41.257 INFO:tasks.rados.rados.0.trial169.stdout:1137: delete oid 19 current snap is 144 2026-02-25T08:46:41.276 INFO:tasks.rados.rados.0.trial169.stdout:1136: expect (ObjNum 499 snap 133 seq_num 499) 2026-02-25T08:46:41.296 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:41.316 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:41.337 INFO:tasks.rados.rados.0.trial169.stdout:1136: done (3 left) 2026-02-25T08:46:41.337 INFO:tasks.rados.rados.0.trial169.stdout:1137: done (2 left) 2026-02-25T08:46:41.337 INFO:tasks.rados.rados.0.trial169.stdout:1138: write (excl) oid 34 current snap is 144 2026-02-25T08:46:41.337 INFO:tasks.rados.rados.0.trial169.stdout:1134: finishing write tid 1 to trial16941936-17 2026-02-25T08:46:41.337 INFO:tasks.rados.rados.0.trial169.stdout:1134: finishing write tid 2 to trial16941936-17 2026-02-25T08:46:41.338 INFO:tasks.rados.rados.0.trial169.stdout:1138: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2026-02-25T08:46:41.342 INFO:tasks.rados.rados.0.trial169.stdout:1138: writing trial16941936-34 from 776622 to 1323639 tid 1 2026-02-25T08:46:41.347 INFO:tasks.rados.rados.0.trial169.stdout:1138: writing trial16941936-34 from 1800933 to 2526451 tid 2 2026-02-25T08:46:41.350 INFO:tasks.rados.rados.0.trial169.stdout:1138: writing trial16941936-34 from 2944601 to 3176622 tid 3 2026-02-25T08:46:41.351 INFO:tasks.rados.rados.0.trial169.stdout:1139: read oid 24 snap -1 2026-02-25T08:46:41.351 INFO:tasks.rados.rados.0.trial169.stdout:1139: expect (ObjNum 519 snap 138 seq_num 519) 2026-02-25T08:46:41.351 INFO:tasks.rados.rados.0.trial169.stdout:1140: read oid 50 snap 129 2026-02-25T08:46:41.351 INFO:tasks.rados.rados.0.trial169.stdout:1140: expect (ObjNum 472 snap 126 seq_num 472) 2026-02-25T08:46:41.351 INFO:tasks.rados.rados.0.trial169.stdout:1141: delete oid 38 current snap is 144 2026-02-25T08:46:41.351 INFO:tasks.rados.rados.0.trial169.stdout:1135: finishing write tid 1 to trial16941936-9 2026-02-25T08:46:41.351 INFO:tasks.rados.rados.0.trial169.stdout:1134: finishing write tid 3 to trial16941936-17 2026-02-25T08:46:41.351 INFO:tasks.rados.rados.0.trial169.stdout:1134: finishing write tid 4 to trial16941936-17 2026-02-25T08:46:41.351 INFO:tasks.rados.rados.0.trial169.stdout:1134: oid 17 updating version 0 to 107 2026-02-25T08:46:41.351 INFO:tasks.rados.rados.0.trial169.stdout:1134: oid 17 version 107 is already newer than 107 2026-02-25T08:46:41.351 INFO:tasks.rados.rados.0.trial169.stdout:1134: oid 17 version 107 is already newer than 107 2026-02-25T08:46:41.352 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 17 v 107 (ObjNum 543 snap 144 seq_num 543) dirty exists 2026-02-25T08:46:41.352 INFO:tasks.rados.rados.0.trial169.stdout:1134: left oid 17 (ObjNum 543 snap 144 seq_num 543) 2026-02-25T08:46:41.352 INFO:tasks.rados.rados.0.trial169.stdout:1135: finishing write tid 2 to trial16941936-9 2026-02-25T08:46:41.352 INFO:tasks.rados.rados.0.trial169.stdout:1135: finishing write tid 3 to trial16941936-9 2026-02-25T08:46:41.352 INFO:tasks.rados.rados.0.trial169.stdout:1135: finishing write tid 4 to trial16941936-9 2026-02-25T08:46:41.352 INFO:tasks.rados.rados.0.trial169.stdout:1135: oid 9 updating version 0 to 174 2026-02-25T08:46:41.352 INFO:tasks.rados.rados.0.trial169.stdout:1135: oid 9 updating version 174 to 176 2026-02-25T08:46:41.352 INFO:tasks.rados.rados.0.trial169.stdout:1135: oid 9 version 176 is already newer than 176 2026-02-25T08:46:41.352 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 9 v 176 (ObjNum 544 snap 144 seq_num 544) dirty exists 2026-02-25T08:46:41.352 INFO:tasks.rados.rados.0.trial169.stdout:1135: left oid 9 (ObjNum 544 snap 144 seq_num 544) 2026-02-25T08:46:41.365 INFO:tasks.rados.rados.0.trial169.stdout:1139: expect (ObjNum 519 snap 138 seq_num 519) 2026-02-25T08:46:41.381 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 801488 2026-02-25T08:46:41.418 INFO:tasks.rados.rados.0.trial169.stdout:1138: finishing write tid 1 to trial16941936-34 2026-02-25T08:46:41.418 INFO:tasks.rados.rados.0.trial169.stdout:1134: done (5 left) 2026-02-25T08:46:41.418 INFO:tasks.rados.rados.0.trial169.stdout:1135: done (4 left) 2026-02-25T08:46:41.418 INFO:tasks.rados.rados.0.trial169.stdout:1139: done (3 left) 2026-02-25T08:46:41.418 INFO:tasks.rados.rados.0.trial169.stdout:1141: done (2 left) 2026-02-25T08:46:41.419 INFO:tasks.rados.rados.0.trial169.stdout:1142: write oid 13 current snap is 144 2026-02-25T08:46:41.419 INFO:tasks.rados.rados.0.trial169.stdout:1142: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2026-02-25T08:46:41.421 INFO:tasks.rados.rados.0.trial169.stdout:1142: writing trial16941936-13 from 410236 to 824104 tid 1 2026-02-25T08:46:41.426 INFO:tasks.rados.rados.0.trial169.stdout:1142: writing trial16941936-13 from 1234250 to 1963437 tid 2 2026-02-25T08:46:41.430 INFO:tasks.rados.rados.0.trial169.stdout:1142: writing trial16941936-13 from 2715800 to 3138863 tid 3 2026-02-25T08:46:41.431 INFO:tasks.rados.rados.0.trial169.stdout:1143: write oid 49 current snap is 144 2026-02-25T08:46:41.431 INFO:tasks.rados.rados.0.trial169.stdout:1143: seq_num 549 ranges {427043=489117,1576379=472845,2626870=600173} 2026-02-25T08:46:41.434 INFO:tasks.rados.rados.0.trial169.stdout:1143: writing trial16941936-49 from 427043 to 916160 tid 1 2026-02-25T08:46:41.439 INFO:tasks.rados.rados.0.trial169.stdout:1143: writing trial16941936-49 from 1576379 to 2049224 tid 2 2026-02-25T08:46:41.444 INFO:tasks.rados.rados.0.trial169.stdout:1143: writing trial16941936-49 from 2626870 to 3227043 tid 3 2026-02-25T08:46:41.444 INFO:tasks.rados.rados.0.trial169.stdout:1144: read oid 14 snap -1 2026-02-25T08:46:41.444 INFO:tasks.rados.rados.0.trial169.stdout:1144: expect (ObjNum 529 snap 142 seq_num 529) 2026-02-25T08:46:41.444 INFO:tasks.rados.rados.0.trial169.stdout:1145: read oid 31 snap -1 2026-02-25T08:46:41.444 INFO:tasks.rados.rados.0.trial169.stdout:1145: expect (ObjNum 513 snap 138 seq_num 513) 2026-02-25T08:46:41.445 INFO:tasks.rados.rados.0.trial169.stdout:1138: finishing write tid 2 to trial16941936-34 2026-02-25T08:46:41.445 INFO:tasks.rados.rados.0.trial169.stdout:1146: read oid 38 snap -1 2026-02-25T08:46:41.445 INFO:tasks.rados.rados.0.trial169.stdout:1146: expect deleted 2026-02-25T08:46:41.445 INFO:tasks.rados.rados.0.trial169.stdout:1147: write oid 25 current snap is 144 2026-02-25T08:46:41.445 INFO:tasks.rados.rados.0.trial169.stdout:1138: finishing write tid 3 to trial16941936-34 2026-02-25T08:46:41.445 INFO:tasks.rados.rados.0.trial169.stdout:1138: finishing write tid 4 to trial16941936-34 2026-02-25T08:46:41.445 INFO:tasks.rados.rados.0.trial169.stdout:1147: seq_num 550 ranges {443850=564366,1634861=732856,2770646=473204} 2026-02-25T08:46:41.447 INFO:tasks.rados.rados.0.trial169.stdout:1147: writing trial16941936-25 from 443850 to 1008216 tid 1 2026-02-25T08:46:41.452 INFO:tasks.rados.rados.0.trial169.stdout:1147: writing trial16941936-25 from 1634861 to 2367717 tid 2 2026-02-25T08:46:41.456 INFO:tasks.rados.rados.0.trial169.stdout:1147: writing trial16941936-25 from 2770646 to 3243850 tid 3 2026-02-25T08:46:41.456 INFO:tasks.rados.rados.0.trial169.stdout:1148: snap_create 2026-02-25T08:46:41.456 INFO:tasks.rados.rados.0.trial169.stdout:1138: finishing write tid 5 to trial16941936-34 2026-02-25T08:46:41.456 INFO:tasks.rados.rados.0.trial169.stdout:1138: oid 34 updating version 0 to 183 2026-02-25T08:46:41.456 INFO:tasks.rados.rados.0.trial169.stdout:1138: oid 34 version 183 is already newer than 180 2026-02-25T08:46:41.456 INFO:tasks.rados.rados.0.trial169.stdout:1138: oid 34 version 183 is already newer than 183 2026-02-25T08:46:41.457 INFO:tasks.rados.rados.0.trial169.stdout:1138: oid 34 version 183 is already newer than 183 2026-02-25T08:46:41.457 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 34 v 183 (ObjNum 546 snap 144 seq_num 546) dirty exists 2026-02-25T08:46:41.457 INFO:tasks.rados.rados.0.trial169.stdout:1138: left oid 34 (ObjNum 546 snap 144 seq_num 546) 2026-02-25T08:46:41.457 INFO:tasks.rados.rados.0.trial169.stdout:1140: expect (ObjNum 472 snap 126 seq_num 472) 2026-02-25T08:46:41.457 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:41.543 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:46:41.612 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:41.615 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:41.616 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:46:41.618 INFO:tasks.rados.rados.0.trial169.stdout:1142: finishing write tid 1 to trial16941936-13 2026-02-25T08:46:41.618 INFO:tasks.rados.rados.0.trial169.stdout:1142: finishing write tid 2 to trial16941936-13 2026-02-25T08:46:41.619 INFO:tasks.rados.rados.0.trial169.stdout:1147: finishing write tid 1 to trial16941936-25 2026-02-25T08:46:41.619 INFO:tasks.rados.rados.0.trial169.stdout:1144: expect (ObjNum 529 snap 142 seq_num 529) 2026-02-25T08:46:41.626 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:41.701 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:41.698+0000 7fd0c085a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:41.703 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:41.702+0000 7fd0c085a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:41.707 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:41.716 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:41.744 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:41.745 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:41.776 INFO:tasks.rados.rados.0.trial169.stdout:1142: finishing write tid 3 to trial16941936-13 2026-02-25T08:46:41.776 INFO:tasks.rados.rados.0.trial169.stdout:1147: finishing write tid 2 to trial16941936-25 2026-02-25T08:46:41.776 INFO:tasks.rados.rados.0.trial169.stdout:1142: finishing write tid 4 to trial16941936-13 2026-02-25T08:46:41.776 INFO:tasks.rados.rados.0.trial169.stdout:1147: finishing write tid 3 to trial16941936-25 2026-02-25T08:46:41.776 INFO:tasks.rados.rados.0.trial169.stdout:1142: finishing write tid 5 to trial16941936-13 2026-02-25T08:46:41.777 INFO:tasks.rados.rados.0.trial169.stdout:1142: oid 13 updating version 0 to 505 2026-02-25T08:46:41.777 INFO:tasks.rados.rados.0.trial169.stdout:1142: oid 13 version 505 is already newer than 505 2026-02-25T08:46:41.777 INFO:tasks.rados.rados.0.trial169.stdout:1142: oid 13 version 505 is already newer than 505 2026-02-25T08:46:41.777 INFO:tasks.rados.rados.0.trial169.stdout:1142: oid 13 version 505 is already newer than 505 2026-02-25T08:46:41.777 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 13 v 505 (ObjNum 548 snap 144 seq_num 548) dirty exists 2026-02-25T08:46:41.777 INFO:tasks.rados.rados.0.trial169.stdout:1142: left oid 13 (ObjNum 548 snap 144 seq_num 548) 2026-02-25T08:46:41.777 INFO:tasks.rados.rados.0.trial169.stdout:1147: finishing write tid 4 to trial16941936-25 2026-02-25T08:46:41.777 INFO:tasks.rados.rados.0.trial169.stdout:1145: expect (ObjNum 513 snap 138 seq_num 513) 2026-02-25T08:46:41.777 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 336496 2026-02-25T08:46:41.857 INFO:tasks.rados.rados.0.trial169.stdout:1147: finishing write tid 5 to trial16941936-25 2026-02-25T08:46:41.857 INFO:tasks.rados.rados.0.trial169.stdout:1147: oid 25 updating version 0 to 120 2026-02-25T08:46:41.857 INFO:tasks.rados.rados.0.trial169.stdout:1147: oid 25 version 120 is already newer than 120 2026-02-25T08:46:41.857 INFO:tasks.rados.rados.0.trial169.stdout:1147: oid 25 version 120 is already newer than 120 2026-02-25T08:46:41.857 INFO:tasks.rados.rados.0.trial169.stdout:1147: oid 25 version 120 is already newer than 120 2026-02-25T08:46:41.857 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 25 v 120 (ObjNum 550 snap 144 seq_num 550) dirty exists 2026-02-25T08:46:41.857 INFO:tasks.rados.rados.0.trial169.stdout:1147: left oid 25 (ObjNum 550 snap 144 seq_num 550) 2026-02-25T08:46:41.857 INFO:tasks.rados.rados.0.trial169.stdout:1143: finishing write tid 1 to trial16941936-49 2026-02-25T08:46:41.858 INFO:tasks.rados.rados.0.trial169.stdout:1143: finishing write tid 2 to trial16941936-49 2026-02-25T08:46:41.858 INFO:tasks.rados.rados.0.trial169.stdout:1143: finishing write tid 3 to trial16941936-49 2026-02-25T08:46:41.858 INFO:tasks.rados.rados.0.trial169.stdout:1143: finishing write tid 4 to trial16941936-49 2026-02-25T08:46:41.858 INFO:tasks.rados.rados.0.trial169.stdout:1143: finishing write tid 5 to trial16941936-49 2026-02-25T08:46:41.858 INFO:tasks.rados.rados.0.trial169.stdout:1143: oid 49 updating version 0 to 510 2026-02-25T08:46:41.858 INFO:tasks.rados.rados.0.trial169.stdout:1143: oid 49 version 510 is already newer than 510 2026-02-25T08:46:41.858 INFO:tasks.rados.rados.0.trial169.stdout:1143: oid 49 version 510 is already newer than 510 2026-02-25T08:46:41.858 INFO:tasks.rados.rados.0.trial169.stdout:1143: oid 49 version 510 is already newer than 510 2026-02-25T08:46:41.858 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 49 v 510 (ObjNum 549 snap 144 seq_num 549) dirty exists 2026-02-25T08:46:41.858 INFO:tasks.rados.rados.0.trial169.stdout:1143: left oid 49 (ObjNum 549 snap 144 seq_num 549) 2026-02-25T08:46:42.191 INFO:tasks.rados.rados.0.trial169.stdout:1138: done (8 left) 2026-02-25T08:46:42.191 INFO:tasks.rados.rados.0.trial169.stdout:1140: done (7 left) 2026-02-25T08:46:42.191 INFO:tasks.rados.rados.0.trial169.stdout:1142: done (6 left) 2026-02-25T08:46:42.191 INFO:tasks.rados.rados.0.trial169.stdout:1143: done (5 left) 2026-02-25T08:46:42.191 INFO:tasks.rados.rados.0.trial169.stdout:1144: done (4 left) 2026-02-25T08:46:42.191 INFO:tasks.rados.rados.0.trial169.stdout:1145: done (3 left) 2026-02-25T08:46:42.191 INFO:tasks.rados.rados.0.trial169.stdout:1146: done (2 left) 2026-02-25T08:46:42.191 INFO:tasks.rados.rados.0.trial169.stdout:1147: done (1 left) 2026-02-25T08:46:42.191 INFO:tasks.rados.rados.0.trial169.stdout:1148: done (0 left) 2026-02-25T08:46:42.191 INFO:tasks.rados.rados.0.trial169.stdout:1149: delete oid 18 current snap is 145 2026-02-25T08:46:42.235 INFO:tasks.rados.rados.0.trial169.stdout:1149: done (0 left) 2026-02-25T08:46:42.235 INFO:tasks.rados.rados.0.trial169.stdout:1150: read oid 8 snap -1 2026-02-25T08:46:42.235 INFO:tasks.rados.rados.0.trial169.stdout:1150: expect (ObjNum 525 snap 141 seq_num 525) 2026-02-25T08:46:42.235 INFO:tasks.rados.rados.0.trial169.stdout:1151: snap_remove snap 135 2026-02-25T08:46:43.192 INFO:tasks.rados.rados.0.trial169.stdout:1151: done (1 left) 2026-02-25T08:46:43.192 INFO:tasks.rados.rados.0.trial169.stdout:1152: delete oid 47 current snap is 145 2026-02-25T08:46:43.192 INFO:tasks.rados.rados.0.trial169.stdout:1150: expect (ObjNum 525 snap 141 seq_num 525) 2026-02-25T08:46:43.199 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2090913 2026-02-25T08:46:43.226 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 639688 2026-02-25T08:46:43.260 INFO:tasks.rados.rados.0.trial169.stdout:1150: done (1 left) 2026-02-25T08:46:43.260 INFO:tasks.rados.rados.0.trial169.stdout:1152: done (0 left) 2026-02-25T08:46:43.260 INFO:tasks.rados.rados.0.trial169.stdout:1153: write oid 27 current snap is 145 2026-02-25T08:46:43.260 INFO:tasks.rados.rados.0.trial169.stdout:1153: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2026-02-25T08:46:43.265 INFO:tasks.rados.rados.0.trial169.stdout:1153: writing trial16941936-27 from 494271 to 1284384 tid 1 2026-02-25T08:46:43.268 INFO:tasks.rados.rados.0.trial169.stdout:1153: writing trial16941936-27 from 1693954 to 2174137 tid 2 2026-02-25T08:46:43.272 INFO:tasks.rados.rados.0.trial169.stdout:1153: writing trial16941936-27 from 2620209 to 3132338 tid 3 2026-02-25T08:46:43.272 INFO:tasks.rados.rados.0.trial169.stdout:1154: read oid 40 snap -1 2026-02-25T08:46:43.272 INFO:tasks.rados.rados.0.trial169.stdout:1154: expect deleted 2026-02-25T08:46:43.272 INFO:tasks.rados.rados.0.trial169.stdout:1155: delete oid 50 current snap is 145 2026-02-25T08:46:43.286 INFO:tasks.rados.rados.0.trial169.stdout:1153: finishing write tid 1 to trial16941936-27 2026-02-25T08:46:43.292 INFO:tasks.rados.rados.0.trial169.stdout:1153: finishing write tid 2 to trial16941936-27 2026-02-25T08:46:43.297 INFO:tasks.rados.rados.0.trial169.stdout:1153: finishing write tid 3 to trial16941936-27 2026-02-25T08:46:43.301 INFO:tasks.rados.rados.0.trial169.stdout:1153: finishing write tid 4 to trial16941936-27 2026-02-25T08:46:43.303 INFO:tasks.rados.rados.0.trial169.stdout:1153: finishing write tid 5 to trial16941936-27 2026-02-25T08:46:43.303 INFO:tasks.rados.rados.0.trial169.stdout:1153: oid 27 updating version 0 to 114 2026-02-25T08:46:43.303 INFO:tasks.rados.rados.0.trial169.stdout:1153: oid 27 version 114 is already newer than 114 2026-02-25T08:46:43.303 INFO:tasks.rados.rados.0.trial169.stdout:1153: oid 27 version 114 is already newer than 114 2026-02-25T08:46:43.303 INFO:tasks.rados.rados.0.trial169.stdout:1153: oid 27 version 114 is already newer than 114 2026-02-25T08:46:43.304 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 27 v 114 (ObjNum 553 snap 145 seq_num 553) dirty exists 2026-02-25T08:46:43.304 INFO:tasks.rados.rados.0.trial169.stdout:1153: left oid 27 (ObjNum 553 snap 145 seq_num 553) 2026-02-25T08:46:43.311 INFO:tasks.rados.rados.0.trial169.stdout:1153: done (2 left) 2026-02-25T08:46:43.311 INFO:tasks.rados.rados.0.trial169.stdout:1154: done (1 left) 2026-02-25T08:46:43.311 INFO:tasks.rados.rados.0.trial169.stdout:1155: done (0 left) 2026-02-25T08:46:43.311 INFO:tasks.rados.rados.0.trial169.stdout:1156: snap_remove snap 136 2026-02-25T08:46:43.746 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:43.826 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:43.824+0000 7facdb25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:43.829 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:43.827+0000 7facdb25a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:43.833 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:43.871 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:43.872 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:44.197 INFO:tasks.rados.rados.0.trial169.stdout:1156: done (0 left) 2026-02-25T08:46:44.197 INFO:tasks.rados.rados.0.trial169.stdout:1157: read oid 5 snap 129 2026-02-25T08:46:44.197 INFO:tasks.rados.rados.0.trial169.stdout:1157: expect (ObjNum 452 snap 122 seq_num 452) 2026-02-25T08:46:44.197 INFO:tasks.rados.rados.0.trial169.stdout:1158: write (excl) oid 20 current snap is 145 2026-02-25T08:46:44.197 INFO:tasks.rados.rados.0.trial169.stdout:1158: seq_num 555 ranges {527885=656964,1927271=483852,3191408=136477} 2026-02-25T08:46:44.201 INFO:tasks.rados.rados.0.trial169.stdout:1158: writing trial16941936-20 from 527885 to 1184849 tid 1 2026-02-25T08:46:44.206 INFO:tasks.rados.rados.0.trial169.stdout:1158: writing trial16941936-20 from 1927271 to 2411123 tid 2 2026-02-25T08:46:44.209 INFO:tasks.rados.rados.0.trial169.stdout:1158: writing trial16941936-20 from 3191408 to 3327885 tid 3 2026-02-25T08:46:44.210 INFO:tasks.rados.rados.0.trial169.stdout:1159: snap_remove snap 144 2026-02-25T08:46:45.200 INFO:tasks.rados.rados.0.trial169.stdout:1159: done (2 left) 2026-02-25T08:46:45.200 INFO:tasks.rados.rados.0.trial169.stdout:1160: read oid 31 snap -1 2026-02-25T08:46:45.200 INFO:tasks.rados.rados.0.trial169.stdout:1160: expect (ObjNum 513 snap 138 seq_num 513) 2026-02-25T08:46:45.200 INFO:tasks.rados.rados.0.trial169.stdout:1161: read oid 7 snap -1 2026-02-25T08:46:45.200 INFO:tasks.rados.rados.0.trial169.stdout:1161: expect (ObjNum 535 snap 142 seq_num 535) 2026-02-25T08:46:45.200 INFO:tasks.rados.rados.0.trial169.stdout:1162: read oid 42 snap -1 2026-02-25T08:46:45.200 INFO:tasks.rados.rados.0.trial169.stdout:1162: expect (ObjNum 524 snap 141 seq_num 524) 2026-02-25T08:46:45.200 INFO:tasks.rados.rados.0.trial169.stdout:1163: write oid 29 current snap is 145 2026-02-25T08:46:45.201 INFO:tasks.rados.rados.0.trial169.stdout:1158: finishing write tid 1 to trial16941936-20 2026-02-25T08:46:45.201 INFO:tasks.rados.rados.0.trial169.stdout:1163: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2026-02-25T08:46:45.204 INFO:tasks.rados.rados.0.trial169.stdout:1163: writing trial16941936-29 from 544692 to 1276905 tid 1 2026-02-25T08:46:45.210 INFO:tasks.rados.rados.0.trial169.stdout:1163: writing trial16941936-29 from 1985753 to 2729616 tid 2 2026-02-25T08:46:45.212 INFO:tasks.rados.rados.0.trial169.stdout:1163: writing trial16941936-29 from 3335184 to 3344692 tid 3 2026-02-25T08:46:45.212 INFO:tasks.rados.rados.0.trial169.stdout:1164: snap_create 2026-02-25T08:46:45.213 INFO:tasks.rados.rados.0.trial169.stdout:1158: finishing write tid 2 to trial16941936-20 2026-02-25T08:46:45.213 INFO:tasks.rados.rados.0.trial169.stdout:1157: expect (ObjNum 452 snap 122 seq_num 452) 2026-02-25T08:46:45.263 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:45.359 INFO:tasks.rados.rados.0.trial169.stdout:1158: finishing write tid 3 to trial16941936-20 2026-02-25T08:46:45.359 INFO:tasks.rados.rados.0.trial169.stdout:1158: finishing write tid 4 to trial16941936-20 2026-02-25T08:46:45.359 INFO:tasks.rados.rados.0.trial169.stdout:1158: finishing write tid 5 to trial16941936-20 2026-02-25T08:46:45.359 INFO:tasks.rados.rados.0.trial169.stdout:1158: oid 20 updating version 0 to 164 2026-02-25T08:46:45.359 INFO:tasks.rados.rados.0.trial169.stdout:1158: oid 20 version 164 is already newer than 164 2026-02-25T08:46:45.360 INFO:tasks.rados.rados.0.trial169.stdout:1158: oid 20 version 164 is already newer than 161 2026-02-25T08:46:45.360 INFO:tasks.rados.rados.0.trial169.stdout:1158: oid 20 version 164 is already newer than 164 2026-02-25T08:46:45.360 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 20 v 164 (ObjNum 555 snap 145 seq_num 555) dirty exists 2026-02-25T08:46:45.360 INFO:tasks.rados.rados.0.trial169.stdout:1158: left oid 20 (ObjNum 555 snap 145 seq_num 555) 2026-02-25T08:46:45.360 INFO:tasks.rados.rados.0.trial169.stdout:1160: expect (ObjNum 513 snap 138 seq_num 513) 2026-02-25T08:46:45.435 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2416495 2026-02-25T08:46:45.440 INFO:tasks.rados.rados.0.trial169.stdout:1162: expect (ObjNum 524 snap 141 seq_num 524) 2026-02-25T08:46:45.442 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 568501 2026-02-25T08:46:45.521 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 602053 2026-02-25T08:46:45.556 INFO:tasks.rados.rados.0.trial169.stdout:1161: expect (ObjNum 535 snap 142 seq_num 535) 2026-02-25T08:46:45.583 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:45.632 INFO:tasks.rados.rados.0.trial169.stdout:1163: finishing write tid 1 to trial16941936-29 2026-02-25T08:46:45.632 INFO:tasks.rados.rados.0.trial169.stdout:1163: finishing write tid 2 to trial16941936-29 2026-02-25T08:46:45.632 INFO:tasks.rados.rados.0.trial169.stdout:1163: finishing write tid 3 to trial16941936-29 2026-02-25T08:46:45.632 INFO:tasks.rados.rados.0.trial169.stdout:1163: finishing write tid 4 to trial16941936-29 2026-02-25T08:46:45.632 INFO:tasks.rados.rados.0.trial169.stdout:1163: finishing write tid 5 to trial16941936-29 2026-02-25T08:46:45.632 INFO:tasks.rados.rados.0.trial169.stdout:1163: oid 29 updating version 0 to 601 2026-02-25T08:46:45.632 INFO:tasks.rados.rados.0.trial169.stdout:1163: oid 29 version 601 is already newer than 601 2026-02-25T08:46:45.632 INFO:tasks.rados.rados.0.trial169.stdout:1163: oid 29 version 601 is already newer than 601 2026-02-25T08:46:45.632 INFO:tasks.rados.rados.0.trial169.stdout:1163: oid 29 version 601 is already newer than 601 2026-02-25T08:46:45.633 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 29 v 601 (ObjNum 556 snap 145 seq_num 556) dirty exists 2026-02-25T08:46:45.633 INFO:tasks.rados.rados.0.trial169.stdout:1163: left oid 29 (ObjNum 556 snap 145 seq_num 556) 2026-02-25T08:46:45.873 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:45.950 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:45.947+0000 7f517f9b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:45.952 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:45.950+0000 7f517f9b8640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:45.955 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:45.994 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:45.995 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:46.204 INFO:tasks.rados.rados.0.trial169.stdout:1157: done (6 left) 2026-02-25T08:46:46.204 INFO:tasks.rados.rados.0.trial169.stdout:1158: done (5 left) 2026-02-25T08:46:46.204 INFO:tasks.rados.rados.0.trial169.stdout:1160: done (4 left) 2026-02-25T08:46:46.204 INFO:tasks.rados.rados.0.trial169.stdout:1161: done (3 left) 2026-02-25T08:46:46.204 INFO:tasks.rados.rados.0.trial169.stdout:1162: done (2 left) 2026-02-25T08:46:46.204 INFO:tasks.rados.rados.0.trial169.stdout:1163: done (1 left) 2026-02-25T08:46:46.204 INFO:tasks.rados.rados.0.trial169.stdout:1164: done (0 left) 2026-02-25T08:46:46.204 INFO:tasks.rados.rados.0.trial169.stdout:1165: snap_create 2026-02-25T08:46:46.617 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:46:46.689 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:46.692 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:46.693 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:46:46.710 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~281s 2026-02-25T08:46:47.205 INFO:tasks.rados.rados.0.trial169.stdout:1165: done (0 left) 2026-02-25T08:46:47.205 INFO:tasks.rados.rados.0.trial169.stdout:1166: snap_create 2026-02-25T08:46:47.996 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:48.075 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:48.072+0000 7fb9e2dcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:48.077 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:48.076+0000 7fb9e2dcc640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:48.081 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:48.119 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:48.119 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:48.166 INFO:tasks.rados.rados.0.trial169.stdout:1166: done (0 left) 2026-02-25T08:46:48.167 INFO:tasks.rados.rados.0.trial169.stdout:1167: read oid 26 snap -1 2026-02-25T08:46:48.167 INFO:tasks.rados.rados.0.trial169.stdout:1167: expect (ObjNum 467 snap 124 seq_num 467) 2026-02-25T08:46:48.167 INFO:tasks.rados.rados.0.trial169.stdout:1168: read oid 29 snap 129 2026-02-25T08:46:48.167 INFO:tasks.rados.rados.0.trial169.stdout:1168: expect deleted 2026-02-25T08:46:48.167 INFO:tasks.rados.rados.0.trial169.stdout:1169: read oid 44 snap -1 2026-02-25T08:46:48.167 INFO:tasks.rados.rados.0.trial169.stdout:1169: expect deleted 2026-02-25T08:46:48.167 INFO:tasks.rados.rados.0.trial169.stdout:1170: snap_create 2026-02-25T08:46:48.216 INFO:tasks.rados.rados.0.trial169.stdout:1167: expect (ObjNum 467 snap 124 seq_num 467) 2026-02-25T08:46:49.168 INFO:tasks.rados.rados.0.trial169.stdout:1167: done (3 left) 2026-02-25T08:46:49.168 INFO:tasks.rados.rados.0.trial169.stdout:1168: done (2 left) 2026-02-25T08:46:49.168 INFO:tasks.rados.rados.0.trial169.stdout:1169: done (1 left) 2026-02-25T08:46:49.168 INFO:tasks.rados.rados.0.trial169.stdout:1170: done (0 left) 2026-02-25T08:46:49.168 INFO:tasks.rados.rados.0.trial169.stdout:1171: read oid 36 snap 131 2026-02-25T08:46:49.168 INFO:tasks.rados.rados.0.trial169.stdout:1171: expect (ObjNum 450 snap 121 seq_num 450) 2026-02-25T08:46:49.168 INFO:tasks.rados.rados.0.trial169.stdout:1172: delete oid 1 current snap is 149 2026-02-25T08:46:49.205 INFO:tasks.rados.rados.0.trial169.stdout:1171: expect (ObjNum 450 snap 121 seq_num 450) 2026-02-25T08:46:49.205 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 303686 2026-02-25T08:46:49.246 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:49.327 INFO:tasks.rados.rados.0.trial169.stdout:1171: done (1 left) 2026-02-25T08:46:49.328 INFO:tasks.rados.rados.0.trial169.stdout:1172: done (0 left) 2026-02-25T08:46:49.328 INFO:tasks.rados.rados.0.trial169.stdout:1173: write (excl) oid 34 current snap is 149 2026-02-25T08:46:49.328 INFO:tasks.rados.rados.0.trial169.stdout:1173: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2026-02-25T08:46:49.331 INFO:tasks.rados.rados.0.trial169.stdout:1173: writing trial16941936-34 from 578306 to 1061017 tid 1 2026-02-25T08:46:49.336 INFO:tasks.rados.rados.0.trial169.stdout:1173: writing trial16941936-34 from 1586364 to 2217543 tid 2 2026-02-25T08:46:49.340 INFO:tasks.rados.rados.0.trial169.stdout:1173: writing trial16941936-34 from 2757324 to 3242166 tid 3 2026-02-25T08:46:49.340 INFO:tasks.rados.rados.0.trial169.stdout:1174: snap_remove snap 147 2026-02-25T08:46:50.120 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:50.170 INFO:tasks.rados.rados.0.trial169.stdout:1174: done (1 left) 2026-02-25T08:46:50.170 INFO:tasks.rados.rados.0.trial169.stdout:1175: read oid 29 snap -1 2026-02-25T08:46:50.170 INFO:tasks.rados.rados.0.trial169.stdout:1175: expect (ObjNum 556 snap 145 seq_num 556) 2026-02-25T08:46:50.170 INFO:tasks.rados.rados.0.trial169.stdout:1176: snap_create 2026-02-25T08:46:50.170 INFO:tasks.rados.rados.0.trial169.stdout:1173: finishing write tid 1 to trial16941936-34 2026-02-25T08:46:50.170 INFO:tasks.rados.rados.0.trial169.stdout:1173: finishing write tid 2 to trial16941936-34 2026-02-25T08:46:50.170 INFO:tasks.rados.rados.0.trial169.stdout:1173: finishing write tid 3 to trial16941936-34 2026-02-25T08:46:50.170 INFO:tasks.rados.rados.0.trial169.stdout:1173: finishing write tid 4 to trial16941936-34 2026-02-25T08:46:50.170 INFO:tasks.rados.rados.0.trial169.stdout:1173: finishing write tid 5 to trial16941936-34 2026-02-25T08:46:50.171 INFO:tasks.rados.rados.0.trial169.stdout:1173: oid 34 updating version 0 to 188 2026-02-25T08:46:50.171 INFO:tasks.rados.rados.0.trial169.stdout:1173: oid 34 version 188 is already newer than 188 2026-02-25T08:46:50.171 INFO:tasks.rados.rados.0.trial169.stdout:1173: oid 34 version 188 is already newer than 188 2026-02-25T08:46:50.171 INFO:tasks.rados.rados.0.trial169.stdout:1173: oid 34 version 188 is already newer than 185 2026-02-25T08:46:50.171 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 34 v 188 (ObjNum 558 snap 149 seq_num 558) dirty exists 2026-02-25T08:46:50.171 INFO:tasks.rados.rados.0.trial169.stdout:1173: left oid 34 (ObjNum 558 snap 149 seq_num 558) 2026-02-25T08:46:50.201 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:50.198+0000 7f4b6ba1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:50.203 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:50.202+0000 7f4b6ba1a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:50.205 INFO:tasks.rados.rados.0.trial169.stdout:1175: expect (ObjNum 556 snap 145 seq_num 556) 2026-02-25T08:46:50.205 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 533673 2026-02-25T08:46:50.207 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:50.246 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:50.247 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:50.269 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:51.172 INFO:tasks.rados.rados.0.trial169.stdout:1173: done (2 left) 2026-02-25T08:46:51.173 INFO:tasks.rados.rados.0.trial169.stdout:1175: done (1 left) 2026-02-25T08:46:51.173 INFO:tasks.rados.rados.0.trial169.stdout:1176: done (0 left) 2026-02-25T08:46:51.173 INFO:tasks.rados.rados.0.trial169.stdout:1177: read oid 1 snap -1 2026-02-25T08:46:51.173 INFO:tasks.rados.rados.0.trial169.stdout:1177: expect deleted 2026-02-25T08:46:51.173 INFO:tasks.rados.rados.0.trial169.stdout:1178: write oid 41 current snap is 150 2026-02-25T08:46:51.173 INFO:tasks.rados.rados.0.trial169.stdout:1178: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2026-02-25T08:46:51.177 INFO:tasks.rados.rados.0.trial169.stdout:1178: writing trial16941936-41 from 595113 to 1153073 tid 1 2026-02-25T08:46:51.181 INFO:tasks.rados.rados.0.trial169.stdout:1178: writing trial16941936-41 from 1644846 to 2136036 tid 2 2026-02-25T08:46:51.186 INFO:tasks.rados.rados.0.trial169.stdout:1178: writing trial16941936-41 from 2784747 to 3395113 tid 3 2026-02-25T08:46:51.186 INFO:tasks.rados.rados.0.trial169.stdout:1179: write oid 21 current snap is 150 2026-02-25T08:46:51.186 INFO:tasks.rados.rados.0.trial169.stdout:1179: seq_num 560 ranges {611920=633209,1703328=634848,2812170=507386} 2026-02-25T08:46:51.190 INFO:tasks.rados.rados.0.trial169.stdout:1179: writing trial16941936-21 from 611920 to 1245129 tid 1 2026-02-25T08:46:51.194 INFO:tasks.rados.rados.0.trial169.stdout:1179: writing trial16941936-21 from 1703328 to 2338176 tid 2 2026-02-25T08:46:51.199 INFO:tasks.rados.rados.0.trial169.stdout:1179: writing trial16941936-21 from 2812170 to 3319556 tid 3 2026-02-25T08:46:51.199 INFO:tasks.rados.rados.0.trial169.stdout:1180: read oid 24 snap 145 2026-02-25T08:46:51.199 INFO:tasks.rados.rados.0.trial169.stdout:1180: expect (ObjNum 519 snap 138 seq_num 519) 2026-02-25T08:46:51.199 INFO:tasks.rados.rados.0.trial169.stdout:1181: write oid 3 current snap is 150 2026-02-25T08:46:51.199 INFO:tasks.rados.rados.0.trial169.stdout:1181: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2026-02-25T08:46:51.203 INFO:tasks.rados.rados.0.trial169.stdout:1181: writing trial16941936-3 from 628727 to 1337185 tid 1 2026-02-25T08:46:51.208 INFO:tasks.rados.rados.0.trial169.stdout:1181: writing trial16941936-3 from 2045457 to 2540316 tid 2 2026-02-25T08:46:51.211 INFO:tasks.rados.rados.0.trial169.stdout:1181: writing trial16941936-3 from 3123240 to 3428727 tid 3 2026-02-25T08:46:51.211 INFO:tasks.rados.rados.0.trial169.stdout:1182: delete oid 33 current snap is 150 2026-02-25T08:46:51.223 INFO:tasks.rados.rados.0.trial169.stdout:1178: finishing write tid 1 to trial16941936-41 2026-02-25T08:46:51.225 INFO:tasks.rados.rados.0.trial169.stdout:1177: done (5 left) 2026-02-25T08:46:51.225 INFO:tasks.rados.rados.0.trial169.stdout:1182: done (4 left) 2026-02-25T08:46:51.225 INFO:tasks.rados.rados.0.trial169.stdout:1183: snap_create 2026-02-25T08:46:51.229 INFO:tasks.rados.rados.0.trial169.stdout:1178: finishing write tid 2 to trial16941936-41 2026-02-25T08:46:51.235 INFO:tasks.rados.rados.0.trial169.stdout:1178: finishing write tid 3 to trial16941936-41 2026-02-25T08:46:51.239 INFO:tasks.rados.rados.0.trial169.stdout:1180: expect (ObjNum 519 snap 138 seq_num 519) 2026-02-25T08:46:51.239 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:51.285 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:51.308 INFO:tasks.rados.rados.0.trial169.stdout:1178: finishing write tid 4 to trial16941936-41 2026-02-25T08:46:51.308 INFO:tasks.rados.rados.0.trial169.stdout:1178: finishing write tid 5 to trial16941936-41 2026-02-25T08:46:51.309 INFO:tasks.rados.rados.0.trial169.stdout:1178: oid 41 updating version 0 to 523 2026-02-25T08:46:51.309 INFO:tasks.rados.rados.0.trial169.stdout:1178: oid 41 version 523 is already newer than 523 2026-02-25T08:46:51.309 INFO:tasks.rados.rados.0.trial169.stdout:1178: oid 41 version 523 is already newer than 523 2026-02-25T08:46:51.309 INFO:tasks.rados.rados.0.trial169.stdout:1178: oid 41 version 523 is already newer than 523 2026-02-25T08:46:51.309 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 41 v 523 (ObjNum 559 snap 150 seq_num 559) dirty exists 2026-02-25T08:46:51.309 INFO:tasks.rados.rados.0.trial169.stdout:1178: left oid 41 (ObjNum 559 snap 150 seq_num 559) 2026-02-25T08:46:51.309 INFO:tasks.rados.rados.0.trial169.stdout:1179: finishing write tid 1 to trial16941936-21 2026-02-25T08:46:51.309 INFO:tasks.rados.rados.0.trial169.stdout:1179: finishing write tid 2 to trial16941936-21 2026-02-25T08:46:51.309 INFO:tasks.rados.rados.0.trial169.stdout:1179: finishing write tid 3 to trial16941936-21 2026-02-25T08:46:51.309 INFO:tasks.rados.rados.0.trial169.stdout:1179: finishing write tid 4 to trial16941936-21 2026-02-25T08:46:51.309 INFO:tasks.rados.rados.0.trial169.stdout:1179: finishing write tid 5 to trial16941936-21 2026-02-25T08:46:51.309 INFO:tasks.rados.rados.0.trial169.stdout:1179: oid 21 updating version 0 to 112 2026-02-25T08:46:51.310 INFO:tasks.rados.rados.0.trial169.stdout:1179: oid 21 version 112 is already newer than 112 2026-02-25T08:46:51.310 INFO:tasks.rados.rados.0.trial169.stdout:1179: oid 21 version 112 is already newer than 112 2026-02-25T08:46:51.310 INFO:tasks.rados.rados.0.trial169.stdout:1179: oid 21 version 112 is already newer than 112 2026-02-25T08:46:51.310 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 21 v 112 (ObjNum 560 snap 150 seq_num 560) dirty exists 2026-02-25T08:46:51.310 INFO:tasks.rados.rados.0.trial169.stdout:1179: left oid 21 (ObjNum 560 snap 150 seq_num 560) 2026-02-25T08:46:51.310 INFO:tasks.rados.rados.0.trial169.stdout:1181: finishing write tid 1 to trial16941936-3 2026-02-25T08:46:51.310 INFO:tasks.rados.rados.0.trial169.stdout:1181: finishing write tid 2 to trial16941936-3 2026-02-25T08:46:51.318 INFO:tasks.rados.rados.0.trial169.stdout:1181: finishing write tid 3 to trial16941936-3 2026-02-25T08:46:51.318 INFO:tasks.rados.rados.0.trial169.stdout:1181: finishing write tid 4 to trial16941936-3 2026-02-25T08:46:51.320 INFO:tasks.rados.rados.0.trial169.stdout:1181: finishing write tid 5 to trial16941936-3 2026-02-25T08:46:51.320 INFO:tasks.rados.rados.0.trial169.stdout:1181: oid 3 updating version 0 to 279 2026-02-25T08:46:51.320 INFO:tasks.rados.rados.0.trial169.stdout:1181: oid 3 version 279 is already newer than 279 2026-02-25T08:46:51.320 INFO:tasks.rados.rados.0.trial169.stdout:1181: oid 3 version 279 is already newer than 279 2026-02-25T08:46:51.320 INFO:tasks.rados.rados.0.trial169.stdout:1181: oid 3 version 279 is already newer than 279 2026-02-25T08:46:51.320 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 3 v 279 (ObjNum 561 snap 150 seq_num 561) dirty exists 2026-02-25T08:46:51.321 INFO:tasks.rados.rados.0.trial169.stdout:1181: left oid 3 (ObjNum 561 snap 150 seq_num 561) 2026-02-25T08:46:51.693 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:46:51.766 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:51.768 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:51.769 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:46:52.178 INFO:tasks.rados.rados.0.trial169.stdout:1178: done (4 left) 2026-02-25T08:46:52.178 INFO:tasks.rados.rados.0.trial169.stdout:1179: done (3 left) 2026-02-25T08:46:52.178 INFO:tasks.rados.rados.0.trial169.stdout:1180: done (2 left) 2026-02-25T08:46:52.179 INFO:tasks.rados.rados.0.trial169.stdout:1181: done (1 left) 2026-02-25T08:46:52.179 INFO:tasks.rados.rados.0.trial169.stdout:1183: done (0 left) 2026-02-25T08:46:52.179 INFO:tasks.rados.rados.0.trial169.stdout:1184: write (excl) oid 31 current snap is 151 2026-02-25T08:46:52.179 INFO:tasks.rados.rados.0.trial169.stdout:1184: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2026-02-25T08:46:52.182 INFO:tasks.rados.rados.0.trial169.stdout:1184: writing trial16941936-31 from 662341 to 1237650 tid 1 2026-02-25T08:46:52.187 INFO:tasks.rados.rados.0.trial169.stdout:1184: writing trial16941936-31 from 1878774 to 2377302 tid 2 2026-02-25T08:46:52.191 INFO:tasks.rados.rados.0.trial169.stdout:1184: writing trial16941936-31 from 2894439 to 3351994 tid 3 2026-02-25T08:46:52.192 INFO:tasks.rados.rados.0.trial169.stdout:1185: read oid 10 snap -1 2026-02-25T08:46:52.192 INFO:tasks.rados.rados.0.trial169.stdout:1185: expect (ObjNum 527 snap 142 seq_num 527) 2026-02-25T08:46:52.192 INFO:tasks.rados.rados.0.trial169.stdout:1186: write oid 50 current snap is 151 2026-02-25T08:46:52.192 INFO:tasks.rados.rados.0.trial169.stdout:1186: seq_num 564 ranges {679148=650558,1937256=642186,3321862=157286} 2026-02-25T08:46:52.196 INFO:tasks.rados.rados.0.trial169.stdout:1186: writing trial16941936-50 from 679148 to 1329706 tid 1 2026-02-25T08:46:52.201 INFO:tasks.rados.rados.0.trial169.stdout:1186: writing trial16941936-50 from 1937256 to 2579442 tid 2 2026-02-25T08:46:52.205 INFO:tasks.rados.rados.0.trial169.stdout:1186: writing trial16941936-50 from 3321862 to 3479148 tid 3 2026-02-25T08:46:52.205 INFO:tasks.rados.rados.0.trial169.stdout:1187: read oid 12 snap -1 2026-02-25T08:46:52.205 INFO:tasks.rados.rados.0.trial169.stdout:1187: expect (ObjNum 346 snap 95 seq_num 346) 2026-02-25T08:46:52.205 INFO:tasks.rados.rados.0.trial169.stdout:1188: snap_remove snap 129 2026-02-25T08:46:52.213 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~286s 2026-02-25T08:46:52.248 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:52.323 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:52.321+0000 7f3ec065a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:52.325 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:52.325+0000 7f3ec065a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:52.329 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:52.368 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:52.369 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:53.182 INFO:tasks.rados.rados.0.trial169.stdout:1188: done (4 left) 2026-02-25T08:46:53.182 INFO:tasks.rados.rados.0.trial169.stdout:1189: read oid 16 snap -1 2026-02-25T08:46:53.182 INFO:tasks.rados.rados.0.trial169.stdout:1189: expect deleted 2026-02-25T08:46:53.182 INFO:tasks.rados.rados.0.trial169.stdout:1190: read oid 39 snap 138 2026-02-25T08:46:53.182 INFO:tasks.rados.rados.0.trial169.stdout:1190: expect deleted 2026-02-25T08:46:53.182 INFO:tasks.rados.rados.0.trial169.stdout:1191: write oid 27 current snap is 151 2026-02-25T08:46:53.182 INFO:tasks.rados.rados.0.trial169.stdout:1191: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2026-02-25T08:46:53.186 INFO:tasks.rados.rados.0.trial169.stdout:1191: writing trial16941936-27 from 695955 to 1421762 tid 1 2026-02-25T08:46:53.190 INFO:tasks.rados.rados.0.trial169.stdout:1191: writing trial16941936-27 from 1879385 to 2381582 tid 2 2026-02-25T08:46:53.194 INFO:tasks.rados.rados.0.trial169.stdout:1191: writing trial16941936-27 from 2949285 to 3429384 tid 3 2026-02-25T08:46:53.194 INFO:tasks.rados.rados.0.trial169.stdout:1192: snap_remove snap 146 2026-02-25T08:46:54.185 INFO:tasks.rados.rados.0.trial169.stdout:1192: done (7 left) 2026-02-25T08:46:54.185 INFO:tasks.rados.rados.0.trial169.stdout:1193: write (excl) oid 3 current snap is 151 2026-02-25T08:46:54.185 INFO:tasks.rados.rados.0.trial169.stdout:1185: expect (ObjNum 527 snap 142 seq_num 527) 2026-02-25T08:46:54.238 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 865419 2026-02-25T08:46:54.255 INFO:tasks.rados.rados.0.trial169.stdout:1186: finishing write tid 1 to trial16941936-50 2026-02-25T08:46:54.255 INFO:tasks.rados.rados.0.trial169.stdout:1187: expect (ObjNum 346 snap 95 seq_num 346) 2026-02-25T08:46:54.283 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 3231408 2026-02-25T08:46:54.296 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1054517 2026-02-25T08:46:54.370 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:54.390 INFO:tasks.rados.rados.0.trial169.stdout:1186: finishing write tid 2 to trial16941936-50 2026-02-25T08:46:54.390 INFO:tasks.rados.rados.0.trial169.stdout:1184: finishing write tid 1 to trial16941936-31 2026-02-25T08:46:54.391 INFO:tasks.rados.rados.0.trial169.stdout:1193: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2026-02-25T08:46:54.395 INFO:tasks.rados.rados.0.trial169.stdout:1193: writing trial16941936-3 from 712762 to 1113818 tid 1 2026-02-25T08:46:54.399 INFO:tasks.rados.rados.0.trial169.stdout:1193: writing trial16941936-3 from 1537867 to 2183722 tid 2 2026-02-25T08:46:54.406 INFO:tasks.rados.rados.0.trial169.stdout:1193: writing trial16941936-3 from 2860355 to 3512762 tid 3 2026-02-25T08:46:54.406 INFO:tasks.rados.rados.0.trial169.stdout:1185: done (7 left) 2026-02-25T08:46:54.406 INFO:tasks.rados.rados.0.trial169.stdout:1187: done (6 left) 2026-02-25T08:46:54.406 INFO:tasks.rados.rados.0.trial169.stdout:1194: read oid 15 snap -1 2026-02-25T08:46:54.406 INFO:tasks.rados.rados.0.trial169.stdout:1194: expect (ObjNum 503 snap 135 seq_num 503) 2026-02-25T08:46:54.406 INFO:tasks.rados.rados.0.trial169.stdout:1195: read oid 5 snap -1 2026-02-25T08:46:54.406 INFO:tasks.rados.rados.0.trial169.stdout:1195: expect (ObjNum 510 snap 137 seq_num 510) 2026-02-25T08:46:54.406 INFO:tasks.rados.rados.0.trial169.stdout:1196: delete oid 13 current snap is 151 2026-02-25T08:46:54.406 INFO:tasks.rados.rados.0.trial169.stdout:1186: finishing write tid 3 to trial16941936-50 2026-02-25T08:46:54.407 INFO:tasks.rados.rados.0.trial169.stdout:1186: finishing write tid 4 to trial16941936-50 2026-02-25T08:46:54.407 INFO:tasks.rados.rados.0.trial169.stdout:1186: finishing write tid 5 to trial16941936-50 2026-02-25T08:46:54.407 INFO:tasks.rados.rados.0.trial169.stdout:1186: oid 50 updating version 0 to 611 2026-02-25T08:46:54.407 INFO:tasks.rados.rados.0.trial169.stdout:1186: oid 50 version 611 is already newer than 611 2026-02-25T08:46:54.407 INFO:tasks.rados.rados.0.trial169.stdout:1186: oid 50 version 611 is already newer than 611 2026-02-25T08:46:54.407 INFO:tasks.rados.rados.0.trial169.stdout:1186: oid 50 version 611 is already newer than 611 2026-02-25T08:46:54.407 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 50 v 611 (ObjNum 564 snap 151 seq_num 564) dirty exists 2026-02-25T08:46:54.407 INFO:tasks.rados.rados.0.trial169.stdout:1186: left oid 50 (ObjNum 564 snap 151 seq_num 564) 2026-02-25T08:46:54.407 INFO:tasks.rados.rados.0.trial169.stdout:1184: finishing write tid 2 to trial16941936-31 2026-02-25T08:46:54.407 INFO:tasks.rados.rados.0.trial169.stdout:1184: finishing write tid 3 to trial16941936-31 2026-02-25T08:46:54.407 INFO:tasks.rados.rados.0.trial169.stdout:1184: finishing write tid 4 to trial16941936-31 2026-02-25T08:46:54.407 INFO:tasks.rados.rados.0.trial169.stdout:1184: finishing write tid 5 to trial16941936-31 2026-02-25T08:46:54.407 INFO:tasks.rados.rados.0.trial169.stdout:1184: oid 31 updating version 0 to 525 2026-02-25T08:46:54.408 INFO:tasks.rados.rados.0.trial169.stdout:1184: oid 31 updating version 525 to 528 2026-02-25T08:46:54.408 INFO:tasks.rados.rados.0.trial169.stdout:1184: oid 31 version 528 is already newer than 528 2026-02-25T08:46:54.408 INFO:tasks.rados.rados.0.trial169.stdout:1184: oid 31 version 528 is already newer than 528 2026-02-25T08:46:54.408 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 31 v 528 (ObjNum 563 snap 151 seq_num 563) dirty exists 2026-02-25T08:46:54.408 INFO:tasks.rados.rados.0.trial169.stdout:1184: left oid 31 (ObjNum 563 snap 151 seq_num 563) 2026-02-25T08:46:54.408 INFO:tasks.rados.rados.0.trial169.stdout:1191: finishing write tid 1 to trial16941936-27 2026-02-25T08:46:54.408 INFO:tasks.rados.rados.0.trial169.stdout:1191: finishing write tid 2 to trial16941936-27 2026-02-25T08:46:54.408 INFO:tasks.rados.rados.0.trial169.stdout:1191: finishing write tid 3 to trial16941936-27 2026-02-25T08:46:54.408 INFO:tasks.rados.rados.0.trial169.stdout:1191: finishing write tid 4 to trial16941936-27 2026-02-25T08:46:54.408 INFO:tasks.rados.rados.0.trial169.stdout:1191: finishing write tid 5 to trial16941936-27 2026-02-25T08:46:54.409 INFO:tasks.rados.rados.0.trial169.stdout:1191: oid 27 updating version 0 to 119 2026-02-25T08:46:54.409 INFO:tasks.rados.rados.0.trial169.stdout:1191: oid 27 version 119 is already newer than 119 2026-02-25T08:46:54.409 INFO:tasks.rados.rados.0.trial169.stdout:1191: oid 27 version 119 is already newer than 119 2026-02-25T08:46:54.409 INFO:tasks.rados.rados.0.trial169.stdout:1191: oid 27 version 119 is already newer than 119 2026-02-25T08:46:54.409 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 27 v 119 (ObjNum 565 snap 151 seq_num 565) dirty exists 2026-02-25T08:46:54.409 INFO:tasks.rados.rados.0.trial169.stdout:1191: left oid 27 (ObjNum 565 snap 151 seq_num 565) 2026-02-25T08:46:54.430 INFO:tasks.rados.rados.0.trial169.stdout:1193: finishing write tid 1 to trial16941936-3 2026-02-25T08:46:54.443 INFO:tasks.rados.rados.0.trial169.stdout:1184: done (8 left) 2026-02-25T08:46:54.443 INFO:tasks.rados.rados.0.trial169.stdout:1186: done (7 left) 2026-02-25T08:46:54.443 INFO:tasks.rados.rados.0.trial169.stdout:1189: done (6 left) 2026-02-25T08:46:54.443 INFO:tasks.rados.rados.0.trial169.stdout:1190: done (5 left) 2026-02-25T08:46:54.443 INFO:tasks.rados.rados.0.trial169.stdout:1191: done (4 left) 2026-02-25T08:46:54.443 INFO:tasks.rados.rados.0.trial169.stdout:1196: done (3 left) 2026-02-25T08:46:54.443 INFO:tasks.rados.rados.0.trial169.stdout:1197: write (excl) oid 38 current snap is 151 2026-02-25T08:46:54.443 INFO:tasks.rados.rados.0.trial169.stdout:1197: seq_num 568 ranges {746376=551554,2054831=649524,3315201=231175} 2026-02-25T08:46:54.448 INFO:tasks.rados.rados.0.trial169.stdout:1197: writing trial16941936-38 from 746376 to 1297930 tid 1 2026-02-25T08:46:54.449 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:54.446+0000 7fab1165a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:54.452 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:54.450+0000 7fab1165a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:54.453 INFO:tasks.rados.rados.0.trial169.stdout:1197: writing trial16941936-38 from 2054831 to 2704355 tid 2 2026-02-25T08:46:54.455 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:54.457 INFO:tasks.rados.rados.0.trial169.stdout:1197: writing trial16941936-38 from 3315201 to 3546376 tid 3 2026-02-25T08:46:54.457 INFO:tasks.rados.rados.0.trial169.stdout:1198: read oid 35 snap 138 2026-02-25T08:46:54.457 INFO:tasks.rados.rados.0.trial169.stdout:1198: expect (ObjNum 516 snap 138 seq_num 516) 2026-02-25T08:46:54.457 INFO:tasks.rados.rados.0.trial169.stdout:1199: write oid 37 current snap is 151 2026-02-25T08:46:54.457 INFO:tasks.rados.rados.0.trial169.stdout:1194: expect (ObjNum 503 snap 135 seq_num 503) 2026-02-25T08:46:54.457 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:54.532 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:54.557 INFO:tasks.rados.rados.0.trial169.stdout:1193: finishing write tid 2 to trial16941936-3 2026-02-25T08:46:54.557 INFO:tasks.rados.rados.0.trial169.stdout:1193: finishing write tid 3 to trial16941936-3 2026-02-25T08:46:54.557 INFO:tasks.rados.rados.0.trial169.stdout:1197: finishing write tid 1 to trial16941936-38 2026-02-25T08:46:54.557 INFO:tasks.rados.rados.0.trial169.stdout:1199: seq_num 569 ranges {763183=626803,1996960=793182,3226271=336912} 2026-02-25T08:46:54.563 INFO:tasks.rados.rados.0.trial169.stdout:1199: writing trial16941936-37 from 763183 to 1389986 tid 1 2026-02-25T08:46:54.568 INFO:tasks.rados.rados.0.trial169.stdout:1199: writing trial16941936-37 from 1996960 to 2790142 tid 2 2026-02-25T08:46:54.572 INFO:tasks.rados.rados.0.trial169.stdout:1199: writing trial16941936-37 from 3226271 to 3563183 tid 3 2026-02-25T08:46:54.572 INFO:tasks.rados.rados.0.trial169.stdout:1194: done (5 left) 2026-02-25T08:46:54.572 INFO:tasks.rados.rados.0.trial169.stdout:1200: read oid 13 snap -1 2026-02-25T08:46:54.572 INFO:tasks.rados.rados.0.trial169.stdout:1200: expect deleted 2026-02-25T08:46:54.572 INFO:tasks.rados.rados.0.trial169.stdout:1201: read oid 14 snap -1 2026-02-25T08:46:54.572 INFO:tasks.rados.rados.0.trial169.stdout:1201: expect (ObjNum 529 snap 142 seq_num 529) 2026-02-25T08:46:54.573 INFO:tasks.rados.rados.0.trial169.stdout:1202: read oid 26 snap -1 2026-02-25T08:46:54.573 INFO:tasks.rados.rados.0.trial169.stdout:1202: expect (ObjNum 467 snap 124 seq_num 467) 2026-02-25T08:46:54.573 INFO:tasks.rados.rados.0.trial169.stdout:1203: snap_remove snap 148 2026-02-25T08:46:55.189 INFO:tasks.rados.rados.0.trial169.stdout:1203: done (8 left) 2026-02-25T08:46:55.190 INFO:tasks.rados.rados.0.trial169.stdout:1204: read oid 45 snap -1 2026-02-25T08:46:55.190 INFO:tasks.rados.rados.0.trial169.stdout:1204: expect deleted 2026-02-25T08:46:55.190 INFO:tasks.rados.rados.0.trial169.stdout:1205: write oid 4 current snap is 151 2026-02-25T08:46:55.190 INFO:tasks.rados.rados.0.trial169.stdout:1193: finishing write tid 4 to trial16941936-3 2026-02-25T08:46:55.190 INFO:tasks.rados.rados.0.trial169.stdout:1205: seq_num 570 ranges {779990=702052,2055442=653193,3253694=326296} 2026-02-25T08:46:55.194 INFO:tasks.rados.rados.0.trial169.stdout:1205: writing trial16941936-4 from 779990 to 1482042 tid 1 2026-02-25T08:46:55.199 INFO:tasks.rados.rados.0.trial169.stdout:1205: writing trial16941936-4 from 2055442 to 2708635 tid 2 2026-02-25T08:46:55.204 INFO:tasks.rados.rados.0.trial169.stdout:1205: writing trial16941936-4 from 3253694 to 3579990 tid 3 2026-02-25T08:46:55.204 INFO:tasks.rados.rados.0.trial169.stdout:1206: write oid 28 current snap is 151 2026-02-25T08:46:55.204 INFO:tasks.rados.rados.0.trial169.stdout:1193: finishing write tid 5 to trial16941936-3 2026-02-25T08:46:55.204 INFO:tasks.rados.rados.0.trial169.stdout:1193: oid 3 updating version 0 to 288 2026-02-25T08:46:55.204 INFO:tasks.rados.rados.0.trial169.stdout:1193: oid 3 version 288 is already newer than 288 2026-02-25T08:46:55.205 INFO:tasks.rados.rados.0.trial169.stdout:1193: oid 3 version 288 is already newer than 285 2026-02-25T08:46:55.205 INFO:tasks.rados.rados.0.trial169.stdout:1193: oid 3 version 288 is already newer than 288 2026-02-25T08:46:55.205 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 3 v 288 (ObjNum 566 snap 151 seq_num 566) dirty exists 2026-02-25T08:46:55.205 INFO:tasks.rados.rados.0.trial169.stdout:1193: left oid 3 (ObjNum 566 snap 151 seq_num 566) 2026-02-25T08:46:55.205 INFO:tasks.rados.rados.0.trial169.stdout:1206: seq_num 571 ranges {796797=493654,1830277=796851,3397470=199327} 2026-02-25T08:46:55.207 INFO:tasks.rados.rados.0.trial169.stdout:1206: writing trial16941936-28 from 796797 to 1290451 tid 1 2026-02-25T08:46:55.212 INFO:tasks.rados.rados.0.trial169.stdout:1206: writing trial16941936-28 from 1830277 to 2627128 tid 2 2026-02-25T08:46:55.217 INFO:tasks.rados.rados.0.trial169.stdout:1206: writing trial16941936-28 from 3397470 to 3596797 tid 3 2026-02-25T08:46:55.217 INFO:tasks.rados.rados.0.trial169.stdout:1193: done (10 left) 2026-02-25T08:46:55.217 INFO:tasks.rados.rados.0.trial169.stdout:1207: snap_create 2026-02-25T08:46:55.217 INFO:tasks.rados.rados.0.trial169.stdout:1195: expect (ObjNum 510 snap 137 seq_num 510) 2026-02-25T08:46:55.217 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:55.255 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:55.316 INFO:tasks.rados.rados.0.trial169.stdout:1197: finishing write tid 2 to trial16941936-38 2026-02-25T08:46:55.317 INFO:tasks.rados.rados.0.trial169.stdout:1197: finishing write tid 3 to trial16941936-38 2026-02-25T08:46:55.317 INFO:tasks.rados.rados.0.trial169.stdout:1197: finishing write tid 4 to trial16941936-38 2026-02-25T08:46:55.317 INFO:tasks.rados.rados.0.trial169.stdout:1197: finishing write tid 5 to trial16941936-38 2026-02-25T08:46:55.317 INFO:tasks.rados.rados.0.trial169.stdout:1197: oid 38 updating version 0 to 128 2026-02-25T08:46:55.317 INFO:tasks.rados.rados.0.trial169.stdout:1197: oid 38 version 128 is already newer than 128 2026-02-25T08:46:55.317 INFO:tasks.rados.rados.0.trial169.stdout:1197: oid 38 version 128 is already newer than 128 2026-02-25T08:46:55.317 INFO:tasks.rados.rados.0.trial169.stdout:1197: oid 38 version 128 is already newer than 125 2026-02-25T08:46:55.317 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 38 v 128 (ObjNum 568 snap 151 seq_num 568) dirty exists 2026-02-25T08:46:55.317 INFO:tasks.rados.rados.0.trial169.stdout:1197: left oid 38 (ObjNum 568 snap 151 seq_num 568) 2026-02-25T08:46:55.318 INFO:tasks.rados.rados.0.trial169.stdout:1198: expect (ObjNum 516 snap 138 seq_num 516) 2026-02-25T08:46:55.351 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1030460 2026-02-25T08:46:55.369 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:55.398 INFO:tasks.rados.rados.0.trial169.stdout:1202: expect (ObjNum 467 snap 124 seq_num 467) 2026-02-25T08:46:55.612 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 3018605 2026-02-25T08:46:55.626 INFO:tasks.rados.rados.0.trial169.stdout:1199: finishing write tid 1 to trial16941936-37 2026-02-25T08:46:55.626 INFO:tasks.rados.rados.0.trial169.stdout:1199: finishing write tid 2 to trial16941936-37 2026-02-25T08:46:55.626 INFO:tasks.rados.rados.0.trial169.stdout:1199: finishing write tid 3 to trial16941936-37 2026-02-25T08:46:55.626 INFO:tasks.rados.rados.0.trial169.stdout:1199: finishing write tid 4 to trial16941936-37 2026-02-25T08:46:55.626 INFO:tasks.rados.rados.0.trial169.stdout:1199: finishing write tid 5 to trial16941936-37 2026-02-25T08:46:55.626 INFO:tasks.rados.rados.0.trial169.stdout:1199: oid 37 updating version 0 to 197 2026-02-25T08:46:55.626 INFO:tasks.rados.rados.0.trial169.stdout:1199: oid 37 version 197 is already newer than 197 2026-02-25T08:46:55.626 INFO:tasks.rados.rados.0.trial169.stdout:1199: oid 37 version 197 is already newer than 197 2026-02-25T08:46:55.627 INFO:tasks.rados.rados.0.trial169.stdout:1199: oid 37 version 197 is already newer than 197 2026-02-25T08:46:55.627 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 37 v 197 (ObjNum 569 snap 151 seq_num 569) dirty exists 2026-02-25T08:46:55.627 INFO:tasks.rados.rados.0.trial169.stdout:1199: left oid 37 (ObjNum 569 snap 151 seq_num 569) 2026-02-25T08:46:55.627 INFO:tasks.rados.rados.0.trial169.stdout:1201: expect (ObjNum 529 snap 142 seq_num 529) 2026-02-25T08:46:55.792 INFO:tasks.rados.rados.0.trial169.stdout:1206: finishing write tid 1 to trial16941936-28 2026-02-25T08:46:55.792 INFO:tasks.rados.rados.0.trial169.stdout:1206: finishing write tid 2 to trial16941936-28 2026-02-25T08:46:55.792 INFO:tasks.rados.rados.0.trial169.stdout:1206: finishing write tid 3 to trial16941936-28 2026-02-25T08:46:55.792 INFO:tasks.rados.rados.0.trial169.stdout:1205: finishing write tid 1 to trial16941936-4 2026-02-25T08:46:55.792 INFO:tasks.rados.rados.0.trial169.stdout:1206: finishing write tid 4 to trial16941936-28 2026-02-25T08:46:55.792 INFO:tasks.rados.rados.0.trial169.stdout:1206: finishing write tid 5 to trial16941936-28 2026-02-25T08:46:55.792 INFO:tasks.rados.rados.0.trial169.stdout:1206: oid 28 updating version 0 to 64 2026-02-25T08:46:55.793 INFO:tasks.rados.rados.0.trial169.stdout:1206: oid 28 version 64 is already newer than 64 2026-02-25T08:46:55.793 INFO:tasks.rados.rados.0.trial169.stdout:1206: oid 28 version 64 is already newer than 64 2026-02-25T08:46:55.793 INFO:tasks.rados.rados.0.trial169.stdout:1206: oid 28 version 64 is already newer than 64 2026-02-25T08:46:55.793 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 28 v 64 (ObjNum 571 snap 151 seq_num 571) dirty exists 2026-02-25T08:46:55.793 INFO:tasks.rados.rados.0.trial169.stdout:1206: left oid 28 (ObjNum 571 snap 151 seq_num 571) 2026-02-25T08:46:55.793 INFO:tasks.rados.rados.0.trial169.stdout:1205: finishing write tid 2 to trial16941936-4 2026-02-25T08:46:55.793 INFO:tasks.rados.rados.0.trial169.stdout:1205: finishing write tid 3 to trial16941936-4 2026-02-25T08:46:55.793 INFO:tasks.rados.rados.0.trial169.stdout:1205: finishing write tid 4 to trial16941936-4 2026-02-25T08:46:55.793 INFO:tasks.rados.rados.0.trial169.stdout:1205: finishing write tid 5 to trial16941936-4 2026-02-25T08:46:55.793 INFO:tasks.rados.rados.0.trial169.stdout:1205: oid 4 updating version 0 to 620 2026-02-25T08:46:55.793 INFO:tasks.rados.rados.0.trial169.stdout:1205: oid 4 version 620 is already newer than 620 2026-02-25T08:46:55.793 INFO:tasks.rados.rados.0.trial169.stdout:1205: oid 4 version 620 is already newer than 620 2026-02-25T08:46:55.793 INFO:tasks.rados.rados.0.trial169.stdout:1205: oid 4 version 620 is already newer than 620 2026-02-25T08:46:55.794 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 4 v 620 (ObjNum 570 snap 151 seq_num 570) dirty exists 2026-02-25T08:46:55.794 INFO:tasks.rados.rados.0.trial169.stdout:1205: left oid 4 (ObjNum 570 snap 151 seq_num 570) 2026-02-25T08:46:56.195 INFO:tasks.rados.rados.0.trial169.stdout:1195: done (10 left) 2026-02-25T08:46:56.195 INFO:tasks.rados.rados.0.trial169.stdout:1197: done (9 left) 2026-02-25T08:46:56.195 INFO:tasks.rados.rados.0.trial169.stdout:1198: done (8 left) 2026-02-25T08:46:56.195 INFO:tasks.rados.rados.0.trial169.stdout:1199: done (7 left) 2026-02-25T08:46:56.195 INFO:tasks.rados.rados.0.trial169.stdout:1200: done (6 left) 2026-02-25T08:46:56.195 INFO:tasks.rados.rados.0.trial169.stdout:1201: done (5 left) 2026-02-25T08:46:56.196 INFO:tasks.rados.rados.0.trial169.stdout:1202: done (4 left) 2026-02-25T08:46:56.196 INFO:tasks.rados.rados.0.trial169.stdout:1204: done (3 left) 2026-02-25T08:46:56.196 INFO:tasks.rados.rados.0.trial169.stdout:1205: done (2 left) 2026-02-25T08:46:56.196 INFO:tasks.rados.rados.0.trial169.stdout:1206: done (1 left) 2026-02-25T08:46:56.196 INFO:tasks.rados.rados.0.trial169.stdout:1207: done (0 left) 2026-02-25T08:46:56.196 INFO:tasks.rados.rados.0.trial169.stdout:1208: read oid 30 snap 124 2026-02-25T08:46:56.196 INFO:tasks.rados.rados.0.trial169.stdout:1208: expect (ObjNum 470 snap 124 seq_num 470) 2026-02-25T08:46:56.196 INFO:tasks.rados.rados.0.trial169.stdout:1209: read oid 50 snap 149 2026-02-25T08:46:56.196 INFO:tasks.rados.rados.0.trial169.stdout:1209: expect deleted 2026-02-25T08:46:56.196 INFO:tasks.rados.rados.0.trial169.stdout:1210: snap_create 2026-02-25T08:46:56.221 INFO:tasks.rados.rados.0.trial169.stdout:1208: expect (ObjNum 470 snap 124 seq_num 470) 2026-02-25T08:46:56.259 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:46:56.325 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 3336216 2026-02-25T08:46:56.534 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:46:56.612 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:56.609+0000 7f1a80fad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:56.614 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:56.613+0000 7f1a80fad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:56.618 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:56.657 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:56.658 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:56.770 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:46:56.841 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:46:56.843 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:46:56.844 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:46:57.199 INFO:tasks.rados.rados.0.trial169.stdout:1208: done (2 left) 2026-02-25T08:46:57.199 INFO:tasks.rados.rados.0.trial169.stdout:1209: done (1 left) 2026-02-25T08:46:57.200 INFO:tasks.rados.rados.0.trial169.stdout:1210: done (0 left) 2026-02-25T08:46:57.200 INFO:tasks.rados.rados.0.trial169.stdout:1211: write oid 21 current snap is 153 2026-02-25T08:46:57.200 INFO:tasks.rados.rados.0.trial169.stdout:1211: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2026-02-25T08:46:57.203 INFO:tasks.rados.rados.0.trial169.stdout:1211: writing trial16941936-21 from 413604 to 982507 tid 1 2026-02-25T08:46:57.207 INFO:tasks.rados.rados.0.trial169.stdout:1211: writing trial16941936-21 from 1488759 to 2145621 tid 2 2026-02-25T08:46:57.212 INFO:tasks.rados.rados.0.trial169.stdout:1211: writing trial16941936-21 from 2624893 to 3216602 tid 3 2026-02-25T08:46:57.212 INFO:tasks.rados.rados.0.trial169.stdout:1212: write (excl) oid 3 current snap is 153 2026-02-25T08:46:57.212 INFO:tasks.rados.rados.0.trial169.stdout:1212: seq_num 573 ranges {430411=644152,1830888=400520,2935963=402981} 2026-02-25T08:46:57.216 INFO:tasks.rados.rados.0.trial169.stdout:1212: writing trial16941936-3 from 430411 to 1074563 tid 1 2026-02-25T08:46:57.220 INFO:tasks.rados.rados.0.trial169.stdout:1212: writing trial16941936-3 from 1830888 to 2231408 tid 2 2026-02-25T08:46:57.224 INFO:tasks.rados.rados.0.trial169.stdout:1212: writing trial16941936-3 from 2935963 to 3338944 tid 3 2026-02-25T08:46:57.225 INFO:tasks.rados.rados.0.trial169.stdout:1213: write (excl) oid 37 current snap is 153 2026-02-25T08:46:57.225 INFO:tasks.rados.rados.0.trial169.stdout:1213: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2026-02-25T08:46:57.229 INFO:tasks.rados.rados.0.trial169.stdout:1213: writing trial16941936-37 from 447218 to 1166619 tid 1 2026-02-25T08:46:57.234 INFO:tasks.rados.rados.0.trial169.stdout:1213: writing trial16941936-37 from 1889370 to 2549901 tid 2 2026-02-25T08:46:57.238 INFO:tasks.rados.rados.0.trial169.stdout:1213: writing trial16941936-37 from 2963386 to 3577639 tid 3 2026-02-25T08:46:57.238 INFO:tasks.rados.rados.0.trial169.stdout:1214: snap_create 2026-02-25T08:46:57.250 INFO:tasks.rados.rados.0.trial169.stdout:1211: finishing write tid 1 to trial16941936-21 2026-02-25T08:46:57.258 INFO:tasks.rados.rados.0.trial169.stdout:1213: finishing write tid 1 to trial16941936-37 2026-02-25T08:46:57.260 INFO:tasks.rados.rados.0.trial169.stdout:1211: finishing write tid 2 to trial16941936-21 2026-02-25T08:46:57.271 INFO:tasks.rados.rados.0.trial169.stdout:1211: finishing write tid 3 to trial16941936-21 2026-02-25T08:46:57.274 INFO:tasks.rados.rados.0.trial169.stdout:1213: finishing write tid 2 to trial16941936-37 2026-02-25T08:46:57.280 INFO:tasks.rados.rados.0.trial169.stdout:1213: finishing write tid 3 to trial16941936-37 2026-02-25T08:46:57.284 INFO:tasks.rados.rados.0.trial169.stdout:1211: finishing write tid 4 to trial16941936-21 2026-02-25T08:46:57.286 INFO:tasks.rados.rados.0.trial169.stdout:1213: finishing write tid 4 to trial16941936-37 2026-02-25T08:46:57.287 INFO:tasks.rados.rados.0.trial169.stdout:1211: finishing write tid 5 to trial16941936-21 2026-02-25T08:46:57.287 INFO:tasks.rados.rados.0.trial169.stdout:1211: oid 21 updating version 0 to 117 2026-02-25T08:46:57.287 INFO:tasks.rados.rados.0.trial169.stdout:1211: oid 21 version 117 is already newer than 117 2026-02-25T08:46:57.287 INFO:tasks.rados.rados.0.trial169.stdout:1211: oid 21 version 117 is already newer than 117 2026-02-25T08:46:57.288 INFO:tasks.rados.rados.0.trial169.stdout:1211: oid 21 version 117 is already newer than 117 2026-02-25T08:46:57.288 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 21 v 117 (ObjNum 572 snap 153 seq_num 572) dirty exists 2026-02-25T08:46:57.288 INFO:tasks.rados.rados.0.trial169.stdout:1211: left oid 21 (ObjNum 572 snap 153 seq_num 572) 2026-02-25T08:46:57.288 INFO:tasks.rados.rados.0.trial169.stdout:1213: finishing write tid 5 to trial16941936-37 2026-02-25T08:46:57.289 INFO:tasks.rados.rados.0.trial169.stdout:1213: oid 37 updating version 0 to 202 2026-02-25T08:46:57.289 INFO:tasks.rados.rados.0.trial169.stdout:1213: oid 37 version 202 is already newer than 199 2026-02-25T08:46:57.289 INFO:tasks.rados.rados.0.trial169.stdout:1213: oid 37 version 202 is already newer than 202 2026-02-25T08:46:57.289 INFO:tasks.rados.rados.0.trial169.stdout:1213: oid 37 version 202 is already newer than 202 2026-02-25T08:46:57.289 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 37 v 202 (ObjNum 574 snap 153 seq_num 574) dirty exists 2026-02-25T08:46:57.289 INFO:tasks.rados.rados.0.trial169.stdout:1213: left oid 37 (ObjNum 574 snap 153 seq_num 574) 2026-02-25T08:46:57.306 INFO:tasks.rados.rados.0.trial169.stdout:1212: finishing write tid 1 to trial16941936-3 2026-02-25T08:46:57.320 INFO:tasks.rados.rados.0.trial169.stdout:1212: finishing write tid 2 to trial16941936-3 2026-02-25T08:46:57.326 INFO:tasks.rados.rados.0.trial169.stdout:1212: finishing write tid 3 to trial16941936-3 2026-02-25T08:46:57.330 INFO:tasks.rados.rados.0.trial169.stdout:1212: finishing write tid 4 to trial16941936-3 2026-02-25T08:46:57.332 INFO:tasks.rados.rados.0.trial169.stdout:1212: finishing write tid 5 to trial16941936-3 2026-02-25T08:46:57.332 INFO:tasks.rados.rados.0.trial169.stdout:1212: oid 3 updating version 0 to 295 2026-02-25T08:46:57.332 INFO:tasks.rados.rados.0.trial169.stdout:1212: oid 3 version 295 is already newer than 295 2026-02-25T08:46:57.332 INFO:tasks.rados.rados.0.trial169.stdout:1212: oid 3 version 295 is already newer than 292 2026-02-25T08:46:57.333 INFO:tasks.rados.rados.0.trial169.stdout:1212: oid 3 version 295 is already newer than 295 2026-02-25T08:46:57.333 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 3 v 295 (ObjNum 573 snap 153 seq_num 573) dirty exists 2026-02-25T08:46:57.333 INFO:tasks.rados.rados.0.trial169.stdout:1212: left oid 3 (ObjNum 573 snap 153 seq_num 573) 2026-02-25T08:46:57.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~292s 2026-02-25T08:46:58.169 INFO:tasks.rados.rados.0.trial169.stdout:1211: done (3 left) 2026-02-25T08:46:58.169 INFO:tasks.rados.rados.0.trial169.stdout:1212: done (2 left) 2026-02-25T08:46:58.169 INFO:tasks.rados.rados.0.trial169.stdout:1213: done (1 left) 2026-02-25T08:46:58.169 INFO:tasks.rados.rados.0.trial169.stdout:1214: done (0 left) 2026-02-25T08:46:58.169 INFO:tasks.rados.rados.0.trial169.stdout:1215: snap_create 2026-02-25T08:46:58.659 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:46:58.737 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:58.734+0000 7f531110b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:58.739 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:46:58.737+0000 7f531110b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:46:58.743 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:46:58.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:46:58.783 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:46:59.170 INFO:tasks.rados.rados.0.trial169.stdout:1215: done (0 left) 2026-02-25T08:46:59.171 INFO:tasks.rados.rados.0.trial169.stdout:1216: write (excl) oid 19 current snap is 155 2026-02-25T08:46:59.171 INFO:tasks.rados.rados.0.trial169.stdout:1216: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2026-02-25T08:46:59.175 INFO:tasks.rados.rados.0.trial169.stdout:1216: writing trial16941936-19 from 464025 to 1258675 tid 1 2026-02-25T08:46:59.180 INFO:tasks.rados.rados.0.trial169.stdout:1216: writing trial16941936-19 from 1947852 to 2352041 tid 2 2026-02-25T08:46:59.184 INFO:tasks.rados.rados.0.trial169.stdout:1216: writing trial16941936-19 from 2990809 to 3416334 tid 3 2026-02-25T08:46:59.184 INFO:tasks.rados.rados.0.trial169.stdout:1217: delete oid 5 current snap is 155 2026-02-25T08:46:59.198 INFO:tasks.rados.rados.0.trial169.stdout:1216: finishing write tid 1 to trial16941936-19 2026-02-25T08:46:59.213 INFO:tasks.rados.rados.0.trial169.stdout:1216: finishing write tid 2 to trial16941936-19 2026-02-25T08:46:59.217 INFO:tasks.rados.rados.0.trial169.stdout:1217: done (1 left) 2026-02-25T08:46:59.217 INFO:tasks.rados.rados.0.trial169.stdout:1218: snap_remove snap 152 2026-02-25T08:47:00.209 INFO:tasks.rados.rados.0.trial169.stdout:1218: done (1 left) 2026-02-25T08:47:00.209 INFO:tasks.rados.rados.0.trial169.stdout:1219: write (excl) oid 25 current snap is 155 2026-02-25T08:47:00.209 INFO:tasks.rados.rados.0.trial169.stdout:1216: finishing write tid 3 to trial16941936-19 2026-02-25T08:47:00.209 INFO:tasks.rados.rados.0.trial169.stdout:1219: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2026-02-25T08:47:00.212 INFO:tasks.rados.rados.0.trial169.stdout:1219: writing trial16941936-25 from 497639 to 1042787 tid 1 2026-02-25T08:47:00.215 INFO:tasks.rados.rados.0.trial169.stdout:1219: writing trial16941936-25 from 1548463 to 1956321 tid 2 2026-02-25T08:47:00.220 INFO:tasks.rados.rados.0.trial169.stdout:1219: writing trial16941936-25 from 2529302 to 3093724 tid 3 2026-02-25T08:47:00.223 INFO:tasks.rados.rados.0.trial169.stdout:1219: writing trial16941936-25 from 3528143 to 3697639 tid 4 2026-02-25T08:47:00.223 INFO:tasks.rados.rados.0.trial169.stdout:1220: read oid 37 snap -1 2026-02-25T08:47:00.223 INFO:tasks.rados.rados.0.trial169.stdout:1220: expect (ObjNum 574 snap 153 seq_num 574) 2026-02-25T08:47:00.223 INFO:tasks.rados.rados.0.trial169.stdout:1221: snap_remove snap 150 2026-02-25T08:47:00.784 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:47:00.863 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:47:00.860+0000 7f20a57ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:47:00.865 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:47:00.863+0000 7f20a57ad640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:47:00.869 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:47:00.909 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:47:00.909 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:47:01.212 INFO:tasks.rados.rados.0.trial169.stdout:1221: done (3 left) 2026-02-25T08:47:01.212 INFO:tasks.rados.rados.0.trial169.stdout:1222: read oid 34 snap -1 2026-02-25T08:47:01.212 INFO:tasks.rados.rados.0.trial169.stdout:1222: expect (ObjNum 558 snap 149 seq_num 558) 2026-02-25T08:47:01.212 INFO:tasks.rados.rados.0.trial169.stdout:1223: write (excl) oid 40 current snap is 155 2026-02-25T08:47:01.212 INFO:tasks.rados.rados.0.trial169.stdout:1216: finishing write tid 4 to trial16941936-19 2026-02-25T08:47:01.212 INFO:tasks.rados.rados.0.trial169.stdout:1216: finishing write tid 5 to trial16941936-19 2026-02-25T08:47:01.212 INFO:tasks.rados.rados.0.trial169.stdout:1216: oid 19 updating version 0 to 192 2026-02-25T08:47:01.213 INFO:tasks.rados.rados.0.trial169.stdout:1216: oid 19 version 192 is already newer than 192 2026-02-25T08:47:01.213 INFO:tasks.rados.rados.0.trial169.stdout:1216: oid 19 version 192 is already newer than 192 2026-02-25T08:47:01.213 INFO:tasks.rados.rados.0.trial169.stdout:1216: oid 19 version 192 is already newer than 189 2026-02-25T08:47:01.213 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 19 v 192 (ObjNum 575 snap 155 seq_num 575) dirty exists 2026-02-25T08:47:01.213 INFO:tasks.rados.rados.0.trial169.stdout:1216: left oid 19 (ObjNum 575 snap 155 seq_num 575) 2026-02-25T08:47:01.213 INFO:tasks.rados.rados.0.trial169.stdout:1223: seq_num 578 ranges {514446=736750,1723298=667869,3073078=641368} 2026-02-25T08:47:01.216 INFO:tasks.rados.rados.0.trial169.stdout:1223: writing trial16941936-40 from 514446 to 1251196 tid 1 2026-02-25T08:47:01.221 INFO:tasks.rados.rados.0.trial169.stdout:1223: writing trial16941936-40 from 1723298 to 2391167 tid 2 2026-02-25T08:47:01.226 INFO:tasks.rados.rados.0.trial169.stdout:1223: writing trial16941936-40 from 3073078 to 3714446 tid 3 2026-02-25T08:47:01.226 INFO:tasks.rados.rados.0.trial169.stdout:1216: done (4 left) 2026-02-25T08:47:01.226 INFO:tasks.rados.rados.0.trial169.stdout:1224: snap_remove snap 141 2026-02-25T08:47:01.846 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:47:01.918 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:01.921 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:01.921 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:47:02.216 INFO:tasks.rados.rados.0.trial169.stdout:1224: done (4 left) 2026-02-25T08:47:02.217 INFO:tasks.rados.rados.0.trial169.stdout:1225: write oid 27 current snap is 155 2026-02-25T08:47:02.217 INFO:tasks.rados.rados.0.trial169.stdout:1225: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2026-02-25T08:47:02.220 INFO:tasks.rados.rados.0.trial169.stdout:1225: writing trial16941936-27 from 531253 to 943252 tid 1 2026-02-25T08:47:02.223 INFO:tasks.rados.rados.0.trial169.stdout:1225: writing trial16941936-27 from 1381780 to 1793307 tid 2 2026-02-25T08:47:02.227 INFO:tasks.rados.rados.0.trial169.stdout:1225: writing trial16941936-27 from 2300501 to 2887467 tid 3 2026-02-25T08:47:02.231 INFO:tasks.rados.rados.0.trial169.stdout:1225: writing trial16941936-27 from 3376974 to 3731253 tid 4 2026-02-25T08:47:02.231 INFO:tasks.rados.rados.0.trial169.stdout:1226: write oid 17 current snap is 155 2026-02-25T08:47:02.231 INFO:tasks.rados.rados.0.trial169.stdout:1226: seq_num 580 ranges {548060=487248,1440262=671538,2844277=798238} 2026-02-25T08:47:02.235 INFO:tasks.rados.rados.0.trial169.stdout:1226: writing trial16941936-17 from 548060 to 1035308 tid 1 2026-02-25T08:47:02.240 INFO:tasks.rados.rados.0.trial169.stdout:1226: writing trial16941936-17 from 1440262 to 2111800 tid 2 2026-02-25T08:47:02.246 INFO:tasks.rados.rados.0.trial169.stdout:1226: writing trial16941936-17 from 2844277 to 3642515 tid 3 2026-02-25T08:47:02.246 INFO:tasks.rados.rados.0.trial169.stdout:1227: write (excl) oid 28 current snap is 155 2026-02-25T08:47:02.246 INFO:tasks.rados.rados.0.trial169.stdout:1220: expect (ObjNum 574 snap 153 seq_num 574) 2026-02-25T08:47:02.246 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:47:02.303 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 914855 2026-02-25T08:47:02.348 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:47:02.401 INFO:tasks.rados.rados.0.trial169.stdout:1219: finishing write tid 1 to trial16941936-25 2026-02-25T08:47:02.401 INFO:tasks.rados.rados.0.trial169.stdout:1219: finishing write tid 2 to trial16941936-25 2026-02-25T08:47:02.401 INFO:tasks.rados.rados.0.trial169.stdout:1227: seq_num 581 ranges {564867=562497,1898744=415196,2755347=725863} 2026-02-25T08:47:02.405 INFO:tasks.rados.rados.0.trial169.stdout:1227: writing trial16941936-28 from 564867 to 1127364 tid 1 2026-02-25T08:47:02.410 INFO:tasks.rados.rados.0.trial169.stdout:1227: writing trial16941936-28 from 1898744 to 2313940 tid 2 2026-02-25T08:47:02.415 INFO:tasks.rados.rados.0.trial169.stdout:1227: writing trial16941936-28 from 2755347 to 3481210 tid 3 2026-02-25T08:47:02.415 INFO:tasks.rados.rados.0.trial169.stdout:1220: done (6 left) 2026-02-25T08:47:02.415 INFO:tasks.rados.rados.0.trial169.stdout:1228: write (excl) oid 49 current snap is 155 2026-02-25T08:47:02.415 INFO:tasks.rados.rados.0.trial169.stdout:1219: finishing write tid 3 to trial16941936-25 2026-02-25T08:47:02.416 INFO:tasks.rados.rados.0.trial169.stdout:1219: finishing write tid 4 to trial16941936-25 2026-02-25T08:47:02.416 INFO:tasks.rados.rados.0.trial169.stdout:1228: seq_num 582 ranges {581674=637746,1840873=558854,3066417=537135} 2026-02-25T08:47:02.419 INFO:tasks.rados.rados.0.trial169.stdout:1228: writing trial16941936-49 from 581674 to 1219420 tid 1 2026-02-25T08:47:02.423 INFO:tasks.rados.rados.0.trial169.stdout:1228: writing trial16941936-49 from 1840873 to 2399727 tid 2 2026-02-25T08:47:02.429 INFO:tasks.rados.rados.0.trial169.stdout:1228: writing trial16941936-49 from 3066417 to 3603552 tid 3 2026-02-25T08:47:02.429 INFO:tasks.rados.rados.0.trial169.stdout:1229: write (excl) oid 35 current snap is 155 2026-02-25T08:47:02.429 INFO:tasks.rados.rados.0.trial169.stdout:1219: finishing write tid 5 to trial16941936-25 2026-02-25T08:47:02.429 INFO:tasks.rados.rados.0.trial169.stdout:1229: seq_num 583 ranges {598481=712995,1899355=418865,3093840=704641} 2026-02-25T08:47:02.433 INFO:tasks.rados.rados.0.trial169.stdout:1229: writing trial16941936-35 from 598481 to 1311476 tid 1 2026-02-25T08:47:02.437 INFO:tasks.rados.rados.0.trial169.stdout:1229: writing trial16941936-35 from 1899355 to 2318220 tid 2 2026-02-25T08:47:02.444 INFO:tasks.rados.rados.0.trial169.stdout:1229: writing trial16941936-35 from 3093840 to 3798481 tid 3 2026-02-25T08:47:02.444 INFO:tasks.rados.rados.0.trial169.stdout:1230: snap_create 2026-02-25T08:47:02.444 INFO:tasks.rados.rados.0.trial169.stdout:1219: finishing write tid 6 to trial16941936-25 2026-02-25T08:47:02.445 INFO:tasks.rados.rados.0.trial169.stdout:1219: oid 25 updating version 0 to 134 2026-02-25T08:47:02.445 INFO:tasks.rados.rados.0.trial169.stdout:1219: oid 25 version 134 is already newer than 134 2026-02-25T08:47:02.445 INFO:tasks.rados.rados.0.trial169.stdout:1219: oid 25 version 134 is already newer than 130 2026-02-25T08:47:02.445 INFO:tasks.rados.rados.0.trial169.stdout:1219: oid 25 version 134 is already newer than 134 2026-02-25T08:47:02.445 INFO:tasks.rados.rados.0.trial169.stdout:1219: oid 25 version 134 is already newer than 134 2026-02-25T08:47:02.445 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 25 v 134 (ObjNum 577 snap 155 seq_num 577) dirty exists 2026-02-25T08:47:02.445 INFO:tasks.rados.rados.0.trial169.stdout:1219: left oid 25 (ObjNum 577 snap 155 seq_num 577) 2026-02-25T08:47:02.445 INFO:tasks.rados.rados.0.trial169.stdout:1223: finishing write tid 1 to trial16941936-40 2026-02-25T08:47:02.445 INFO:tasks.rados.rados.0.trial169.stdout:1223: finishing write tid 2 to trial16941936-40 2026-02-25T08:47:02.445 INFO:tasks.rados.rados.0.trial169.stdout:1222: expect (ObjNum 558 snap 149 seq_num 558) 2026-02-25T08:47:02.448 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1032274 2026-02-25T08:47:02.609 INFO:tasks.rados.rados.0.trial169.stdout:1223: finishing write tid 3 to trial16941936-40 2026-02-25T08:47:02.609 INFO:tasks.rados.rados.0.trial169.stdout:1223: finishing write tid 4 to trial16941936-40 2026-02-25T08:47:02.609 INFO:tasks.rados.rados.0.trial169.stdout:1223: finishing write tid 5 to trial16941936-40 2026-02-25T08:47:02.609 INFO:tasks.rados.rados.0.trial169.stdout:1223: oid 40 updating version 0 to 626 2026-02-25T08:47:02.609 INFO:tasks.rados.rados.0.trial169.stdout:1223: oid 40 version 626 is already newer than 626 2026-02-25T08:47:02.609 INFO:tasks.rados.rados.0.trial169.stdout:1223: oid 40 version 626 is already newer than 621 2026-02-25T08:47:02.609 INFO:tasks.rados.rados.0.trial169.stdout:1223: oid 40 version 626 is already newer than 626 2026-02-25T08:47:02.609 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 40 v 626 (ObjNum 578 snap 155 seq_num 578) dirty exists 2026-02-25T08:47:02.610 INFO:tasks.rados.rados.0.trial169.stdout:1223: left oid 40 (ObjNum 578 snap 155 seq_num 578) 2026-02-25T08:47:02.610 INFO:tasks.rados.rados.0.trial169.stdout:1226: finishing write tid 1 to trial16941936-17 2026-02-25T08:47:02.610 INFO:tasks.rados.rados.0.trial169.stdout:1226: finishing write tid 2 to trial16941936-17 2026-02-25T08:47:02.610 INFO:tasks.rados.rados.0.trial169.stdout:1226: finishing write tid 3 to trial16941936-17 2026-02-25T08:47:02.610 INFO:tasks.rados.rados.0.trial169.stdout:1226: finishing write tid 4 to trial16941936-17 2026-02-25T08:47:02.610 INFO:tasks.rados.rados.0.trial169.stdout:1226: finishing write tid 5 to trial16941936-17 2026-02-25T08:47:02.610 INFO:tasks.rados.rados.0.trial169.stdout:1226: oid 17 updating version 0 to 122 2026-02-25T08:47:02.610 INFO:tasks.rados.rados.0.trial169.stdout:1226: oid 17 version 122 is already newer than 122 2026-02-25T08:47:02.610 INFO:tasks.rados.rados.0.trial169.stdout:1226: oid 17 version 122 is already newer than 122 2026-02-25T08:47:02.610 INFO:tasks.rados.rados.0.trial169.stdout:1226: oid 17 version 122 is already newer than 122 2026-02-25T08:47:02.610 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 17 v 122 (ObjNum 580 snap 155 seq_num 580) dirty exists 2026-02-25T08:47:02.611 INFO:tasks.rados.rados.0.trial169.stdout:1226: left oid 17 (ObjNum 580 snap 155 seq_num 580) 2026-02-25T08:47:02.611 INFO:tasks.rados.rados.0.trial169.stdout:1225: finishing write tid 1 to trial16941936-27 2026-02-25T08:47:02.611 INFO:tasks.rados.rados.0.trial169.stdout:1225: finishing write tid 2 to trial16941936-27 2026-02-25T08:47:02.611 INFO:tasks.rados.rados.0.trial169.stdout:1225: finishing write tid 3 to trial16941936-27 2026-02-25T08:47:02.611 INFO:tasks.rados.rados.0.trial169.stdout:1225: finishing write tid 4 to trial16941936-27 2026-02-25T08:47:02.611 INFO:tasks.rados.rados.0.trial169.stdout:1225: finishing write tid 5 to trial16941936-27 2026-02-25T08:47:02.611 INFO:tasks.rados.rados.0.trial169.stdout:1225: finishing write tid 6 to trial16941936-27 2026-02-25T08:47:02.611 INFO:tasks.rados.rados.0.trial169.stdout:1225: oid 27 updating version 0 to 125 2026-02-25T08:47:02.611 INFO:tasks.rados.rados.0.trial169.stdout:1225: oid 27 version 125 is already newer than 125 2026-02-25T08:47:02.611 INFO:tasks.rados.rados.0.trial169.stdout:1225: oid 27 version 125 is already newer than 125 2026-02-25T08:47:02.611 INFO:tasks.rados.rados.0.trial169.stdout:1225: oid 27 version 125 is already newer than 125 2026-02-25T08:47:02.611 INFO:tasks.rados.rados.0.trial169.stdout:1225: oid 27 version 125 is already newer than 125 2026-02-25T08:47:02.611 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 27 v 125 (ObjNum 579 snap 155 seq_num 579) dirty exists 2026-02-25T08:47:02.612 INFO:tasks.rados.rados.0.trial169.stdout:1225: left oid 27 (ObjNum 579 snap 155 seq_num 579) 2026-02-25T08:47:02.612 INFO:tasks.rados.rados.0.trial169.stdout:1227: finishing write tid 1 to trial16941936-28 2026-02-25T08:47:02.612 INFO:tasks.rados.rados.0.trial169.stdout:1228: finishing write tid 1 to trial16941936-49 2026-02-25T08:47:02.612 INFO:tasks.rados.rados.0.trial169.stdout:1227: finishing write tid 2 to trial16941936-28 2026-02-25T08:47:02.612 INFO:tasks.rados.rados.0.trial169.stdout:1227: finishing write tid 3 to trial16941936-28 2026-02-25T08:47:02.612 INFO:tasks.rados.rados.0.trial169.stdout:1227: finishing write tid 4 to trial16941936-28 2026-02-25T08:47:02.612 INFO:tasks.rados.rados.0.trial169.stdout:1227: finishing write tid 5 to trial16941936-28 2026-02-25T08:47:02.612 INFO:tasks.rados.rados.0.trial169.stdout:1227: oid 28 updating version 0 to 69 2026-02-25T08:47:02.612 INFO:tasks.rados.rados.0.trial169.stdout:1227: oid 28 version 69 is already newer than 66 2026-02-25T08:47:02.612 INFO:tasks.rados.rados.0.trial169.stdout:1227: oid 28 version 69 is already newer than 69 2026-02-25T08:47:02.612 INFO:tasks.rados.rados.0.trial169.stdout:1227: oid 28 version 69 is already newer than 69 2026-02-25T08:47:02.612 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 28 v 69 (ObjNum 581 snap 155 seq_num 581) dirty exists 2026-02-25T08:47:02.613 INFO:tasks.rados.rados.0.trial169.stdout:1227: left oid 28 (ObjNum 581 snap 155 seq_num 581) 2026-02-25T08:47:02.613 INFO:tasks.rados.rados.0.trial169.stdout:1228: finishing write tid 2 to trial16941936-49 2026-02-25T08:47:02.613 INFO:tasks.rados.rados.0.trial169.stdout:1229: finishing write tid 1 to trial16941936-35 2026-02-25T08:47:02.613 INFO:tasks.rados.rados.0.trial169.stdout:1228: finishing write tid 3 to trial16941936-49 2026-02-25T08:47:02.613 INFO:tasks.rados.rados.0.trial169.stdout:1228: finishing write tid 4 to trial16941936-49 2026-02-25T08:47:02.613 INFO:tasks.rados.rados.0.trial169.stdout:1228: finishing write tid 5 to trial16941936-49 2026-02-25T08:47:02.613 INFO:tasks.rados.rados.0.trial169.stdout:1228: oid 49 updating version 0 to 549 2026-02-25T08:47:02.613 INFO:tasks.rados.rados.0.trial169.stdout:1228: oid 49 version 549 is already newer than 549 2026-02-25T08:47:02.613 INFO:tasks.rados.rados.0.trial169.stdout:1228: oid 49 version 549 is already newer than 549 2026-02-25T08:47:02.613 INFO:tasks.rados.rados.0.trial169.stdout:1228: oid 49 version 549 is already newer than 546 2026-02-25T08:47:02.613 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 49 v 549 (ObjNum 582 snap 155 seq_num 582) dirty exists 2026-02-25T08:47:02.613 INFO:tasks.rados.rados.0.trial169.stdout:1228: left oid 49 (ObjNum 582 snap 155 seq_num 582) 2026-02-25T08:47:02.613 INFO:tasks.rados.rados.0.trial169.stdout:1229: finishing write tid 2 to trial16941936-35 2026-02-25T08:47:02.614 INFO:tasks.rados.rados.0.trial169.stdout:1229: finishing write tid 3 to trial16941936-35 2026-02-25T08:47:02.614 INFO:tasks.rados.rados.0.trial169.stdout:1229: finishing write tid 4 to trial16941936-35 2026-02-25T08:47:02.614 INFO:tasks.rados.rados.0.trial169.stdout:1229: finishing write tid 5 to trial16941936-35 2026-02-25T08:47:02.614 INFO:tasks.rados.rados.0.trial169.stdout:1229: oid 35 updating version 0 to 628 2026-02-25T08:47:02.614 INFO:tasks.rados.rados.0.trial169.stdout:1229: oid 35 updating version 628 to 631 2026-02-25T08:47:02.614 INFO:tasks.rados.rados.0.trial169.stdout:1229: oid 35 version 631 is already newer than 631 2026-02-25T08:47:02.614 INFO:tasks.rados.rados.0.trial169.stdout:1229: oid 35 version 631 is already newer than 631 2026-02-25T08:47:02.614 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 35 v 631 (ObjNum 583 snap 155 seq_num 583) dirty exists 2026-02-25T08:47:02.614 INFO:tasks.rados.rados.0.trial169.stdout:1229: left oid 35 (ObjNum 583 snap 155 seq_num 583) 2026-02-25T08:47:02.910 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:47:02.988 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:47:02.985+0000 7fb48d35b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:47:02.990 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:47:02.988+0000 7fb48d35b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:47:02.994 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:47:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:47:03.033 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:47:03.218 INFO:tasks.rados.rados.0.trial169.stdout:1219: done (8 left) 2026-02-25T08:47:03.218 INFO:tasks.rados.rados.0.trial169.stdout:1222: done (7 left) 2026-02-25T08:47:03.218 INFO:tasks.rados.rados.0.trial169.stdout:1223: done (6 left) 2026-02-25T08:47:03.218 INFO:tasks.rados.rados.0.trial169.stdout:1225: done (5 left) 2026-02-25T08:47:03.218 INFO:tasks.rados.rados.0.trial169.stdout:1226: done (4 left) 2026-02-25T08:47:03.218 INFO:tasks.rados.rados.0.trial169.stdout:1227: done (3 left) 2026-02-25T08:47:03.218 INFO:tasks.rados.rados.0.trial169.stdout:1228: done (2 left) 2026-02-25T08:47:03.218 INFO:tasks.rados.rados.0.trial169.stdout:1229: done (1 left) 2026-02-25T08:47:03.218 INFO:tasks.rados.rados.0.trial169.stdout:1230: done (0 left) 2026-02-25T08:47:03.218 INFO:tasks.rados.rados.0.trial169.stdout:1231: write (excl) oid 3 current snap is 156 2026-02-25T08:47:03.219 INFO:tasks.rados.rados.0.trial169.stdout:1231: seq_num 584 ranges {615288=788244,1957837=562523,3121263=559679} 2026-02-25T08:47:03.221 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~297s 2026-02-25T08:47:03.223 INFO:tasks.rados.rados.0.trial169.stdout:1231: writing trial16941936-3 from 615288 to 1403532 tid 1 2026-02-25T08:47:03.227 INFO:tasks.rados.rados.0.trial169.stdout:1231: writing trial16941936-3 from 1957837 to 2520360 tid 2 2026-02-25T08:47:03.232 INFO:tasks.rados.rados.0.trial169.stdout:1231: writing trial16941936-3 from 3121263 to 3680942 tid 3 2026-02-25T08:47:03.232 INFO:tasks.rados.rados.0.trial169.stdout:1232: delete oid 15 current snap is 156 2026-02-25T08:47:03.284 INFO:tasks.rados.rados.0.trial169.stdout:1232: done (1 left) 2026-02-25T08:47:03.284 INFO:tasks.rados.rados.0.trial169.stdout:1233: write oid 30 current snap is 156 2026-02-25T08:47:03.284 INFO:tasks.rados.rados.0.trial169.stdout:1233: seq_num 586 ranges {648902=655095,2074801=566192,3176109=672793} 2026-02-25T08:47:03.289 INFO:tasks.rados.rados.0.trial169.stdout:1233: writing trial16941936-30 from 648902 to 1303997 tid 1 2026-02-25T08:47:03.294 INFO:tasks.rados.rados.0.trial169.stdout:1233: writing trial16941936-30 from 2074801 to 2640993 tid 2 2026-02-25T08:47:03.299 INFO:tasks.rados.rados.0.trial169.stdout:1233: writing trial16941936-30 from 3176109 to 3848902 tid 3 2026-02-25T08:47:03.299 INFO:tasks.rados.rados.0.trial169.stdout:1234: delete oid 10 current snap is 156 2026-02-25T08:47:03.312 INFO:tasks.rados.rados.0.trial169.stdout:1233: finishing write tid 1 to trial16941936-30 2026-02-25T08:47:03.319 INFO:tasks.rados.rados.0.trial169.stdout:1233: finishing write tid 2 to trial16941936-30 2026-02-25T08:47:03.321 INFO:tasks.rados.rados.0.trial169.stdout:1231: finishing write tid 1 to trial16941936-3 2026-02-25T08:47:03.325 INFO:tasks.rados.rados.0.trial169.stdout:1233: finishing write tid 3 to trial16941936-30 2026-02-25T08:47:03.328 INFO:tasks.rados.rados.0.trial169.stdout:1233: finishing write tid 4 to trial16941936-30 2026-02-25T08:47:03.329 INFO:tasks.rados.rados.0.trial169.stdout:1234: done (2 left) 2026-02-25T08:47:03.329 INFO:tasks.rados.rados.0.trial169.stdout:1235: snap_remove snap 151 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1235: done (2 left) 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1236: read oid 10 snap -1 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1236: expect deleted 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1237: snap_create 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1233: finishing write tid 5 to trial16941936-30 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1233: oid 30 updating version 0 to 168 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1233: oid 30 version 168 is already newer than 168 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1233: oid 30 version 168 is already newer than 168 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1233: oid 30 version 168 is already newer than 168 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 30 v 168 (ObjNum 586 snap 156 seq_num 586) dirty exists 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1233: left oid 30 (ObjNum 586 snap 156 seq_num 586) 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1231: finishing write tid 2 to trial16941936-3 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1231: finishing write tid 3 to trial16941936-3 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1231: finishing write tid 4 to trial16941936-3 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1231: finishing write tid 5 to trial16941936-3 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1231: oid 3 updating version 0 to 310 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1231: oid 3 version 310 is already newer than 307 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1231: oid 3 version 310 is already newer than 310 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1231: oid 3 version 310 is already newer than 310 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 3 v 310 (ObjNum 584 snap 156 seq_num 584) dirty exists 2026-02-25T08:47:04.222 INFO:tasks.rados.rados.0.trial169.stdout:1231: left oid 3 (ObjNum 584 snap 156 seq_num 584) 2026-02-25T08:47:05.034 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2026-02-25T08:47:05.112 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:47:05.110+0000 7f902965a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:47:05.115 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:47:05.113+0000 7f902965a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:47:05.118 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:47:05.158 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:47:05.158 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:47:05.223 INFO:tasks.rados.rados.0.trial169.stdout:1231: done (3 left) 2026-02-25T08:47:05.223 INFO:tasks.rados.rados.0.trial169.stdout:1233: done (2 left) 2026-02-25T08:47:05.223 INFO:tasks.rados.rados.0.trial169.stdout:1236: done (1 left) 2026-02-25T08:47:05.224 INFO:tasks.rados.rados.0.trial169.stdout:1237: done (0 left) 2026-02-25T08:47:05.224 INFO:tasks.rados.rados.0.trial169.stdout:1238: read oid 35 snap 145 2026-02-25T08:47:05.224 INFO:tasks.rados.rados.0.trial169.stdout:1238: expect (ObjNum 516 snap 138 seq_num 516) 2026-02-25T08:47:05.224 INFO:tasks.rados.rados.0.trial169.stdout:1239: delete oid 7 current snap is 157 2026-02-25T08:47:05.259 INFO:tasks.rados.rados.0.trial169.stdout:1238: expect (ObjNum 516 snap 138 seq_num 516) 2026-02-25T08:47:05.265 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 761894 2026-02-25T08:47:05.323 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 2218811 2026-02-25T08:47:05.326 INFO:tasks.rados.rados.0.trial169.stdout:1238: done (1 left) 2026-02-25T08:47:05.326 INFO:tasks.rados.rados.0.trial169.stdout:1239: done (0 left) 2026-02-25T08:47:05.327 INFO:tasks.rados.rados.0.trial169.stdout:1240: write oid 43 current snap is 157 2026-02-25T08:47:05.327 INFO:tasks.rados.rados.0.trial169.stdout:1240: seq_num 589 ranges {699323=480842,1850247=429872,2858378=532392} 2026-02-25T08:47:05.331 INFO:tasks.rados.rados.0.trial169.stdout:1240: writing trial16941936-43 from 699323 to 1180165 tid 1 2026-02-25T08:47:05.335 INFO:tasks.rados.rados.0.trial169.stdout:1240: writing trial16941936-43 from 1850247 to 2280119 tid 2 2026-02-25T08:47:05.339 INFO:tasks.rados.rados.0.trial169.stdout:1240: writing trial16941936-43 from 2858378 to 3390770 tid 3 2026-02-25T08:47:05.339 INFO:tasks.rados.rados.0.trial169.stdout:1241: snap_remove snap 154 2026-02-25T08:47:06.231 INFO:tasks.rados.rados.0.trial169.stdout:1241: done (1 left) 2026-02-25T08:47:06.231 INFO:tasks.rados.rados.0.trial169.stdout:1242: read oid 36 snap -1 2026-02-25T08:47:06.231 INFO:tasks.rados.rados.0.trial169.stdout:1242: expect (ObjNum 450 snap 121 seq_num 450) 2026-02-25T08:47:06.231 INFO:tasks.rados.rados.0.trial169.stdout:1240: finishing write tid 1 to trial16941936-43 2026-02-25T08:47:06.231 INFO:tasks.rados.rados.0.trial169.stdout:1243: read oid 9 snap 155 2026-02-25T08:47:06.231 INFO:tasks.rados.rados.0.trial169.stdout:1243: expect (ObjNum 544 snap 144 seq_num 544) 2026-02-25T08:47:06.231 INFO:tasks.rados.rados.0.trial169.stdout:1244: write oid 46 current snap is 157 2026-02-25T08:47:06.231 INFO:tasks.rados.rados.0.trial169.stdout:1240: finishing write tid 2 to trial16941936-43 2026-02-25T08:47:06.231 INFO:tasks.rados.rados.0.trial169.stdout:1240: finishing write tid 3 to trial16941936-43 2026-02-25T08:47:06.231 INFO:tasks.rados.rados.0.trial169.stdout:1240: finishing write tid 4 to trial16941936-43 2026-02-25T08:47:06.231 INFO:tasks.rados.rados.0.trial169.stdout:1244: seq_num 590 ranges {716130=556091,1792376=573530,2769448=460017,3821956=94174} 2026-02-25T08:47:06.235 INFO:tasks.rados.rados.0.trial169.stdout:1244: writing trial16941936-46 from 716130 to 1272221 tid 1 2026-02-25T08:47:06.239 INFO:tasks.rados.rados.0.trial169.stdout:1244: writing trial16941936-46 from 1792376 to 2365906 tid 2 2026-02-25T08:47:06.243 INFO:tasks.rados.rados.0.trial169.stdout:1244: writing trial16941936-46 from 2769448 to 3229465 tid 3 2026-02-25T08:47:06.245 INFO:tasks.rados.rados.0.trial169.stdout:1244: writing trial16941936-46 from 3821956 to 3916130 tid 4 2026-02-25T08:47:06.245 INFO:tasks.rados.rados.0.trial169.stdout:1245: write oid 45 current snap is 157 2026-02-25T08:47:06.245 INFO:tasks.rados.rados.0.trial169.stdout:1240: finishing write tid 5 to trial16941936-43 2026-02-25T08:47:06.245 INFO:tasks.rados.rados.0.trial169.stdout:1240: oid 43 updating version 0 to 130 2026-02-25T08:47:06.245 INFO:tasks.rados.rados.0.trial169.stdout:1240: oid 43 version 130 is already newer than 130 2026-02-25T08:47:06.246 INFO:tasks.rados.rados.0.trial169.stdout:1240: oid 43 version 130 is already newer than 130 2026-02-25T08:47:06.246 INFO:tasks.rados.rados.0.trial169.stdout:1240: oid 43 version 130 is already newer than 130 2026-02-25T08:47:06.246 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 43 v 130 (ObjNum 589 snap 157 seq_num 589) dirty exists 2026-02-25T08:47:06.246 INFO:tasks.rados.rados.0.trial169.stdout:1240: left oid 43 (ObjNum 589 snap 157 seq_num 589) 2026-02-25T08:47:06.246 INFO:tasks.rados.rados.0.trial169.stdout:1245: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2026-02-25T08:47:06.250 INFO:tasks.rados.rados.0.trial169.stdout:1245: writing trial16941936-45 from 732937 to 1364277 tid 1 2026-02-25T08:47:06.254 INFO:tasks.rados.rados.0.trial169.stdout:1245: writing trial16941936-45 from 1850858 to 2284399 tid 2 2026-02-25T08:47:06.258 INFO:tasks.rados.rados.0.trial169.stdout:1245: writing trial16941936-45 from 2796871 to 3468160 tid 3 2026-02-25T08:47:06.260 INFO:tasks.rados.rados.0.trial169.stdout:1245: writing trial16941936-45 from 3888195 to 3932937 tid 4 2026-02-25T08:47:06.260 INFO:tasks.rados.rados.0.trial169.stdout:1240: done (4 left) 2026-02-25T08:47:06.260 INFO:tasks.rados.rados.0.trial169.stdout:1246: delete oid 38 current snap is 157 2026-02-25T08:47:06.260 INFO:tasks.rados.rados.0.trial169.stdout:1243: expect (ObjNum 544 snap 144 seq_num 544) 2026-02-25T08:47:06.323 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 1271073 2026-02-25T08:47:06.347 INFO:tasks.rados.rados.0.trial169.stdout:1243: done (4 left) 2026-02-25T08:47:06.347 INFO:tasks.rados.rados.0.trial169.stdout:1246: done (3 left) 2026-02-25T08:47:06.347 INFO:tasks.rados.rados.0.trial169.stdout:1247: snap_remove snap 138 2026-02-25T08:47:06.922 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:47:06.996 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:07.000 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:47:07.159 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2026-02-25T08:47:07.230 INFO:tasks.rados.rados.0.trial169.stdout:1247: done (3 left) 2026-02-25T08:47:07.230 INFO:tasks.rados.rados.0.trial169.stdout:1248: delete oid 23 current snap is 157 2026-02-25T08:47:07.230 INFO:tasks.rados.rados.0.trial169.stdout:1242: expect (ObjNum 450 snap 121 seq_num 450) 2026-02-25T08:47:07.230 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 0 2026-02-25T08:47:07.240 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:47:07.237+0000 7f175f05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:47:07.242 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:47:07.240+0000 7f175f05a640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:47:07.245 INFO:teuthology.orchestra.run.trial169.stderr:Error ENXIO: problem getting command descriptions from osd.0 2026-02-25T08:47:07.272 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 342695 2026-02-25T08:47:07.297 DEBUG:teuthology.orchestra.run:got remote process result: 6 2026-02-25T08:47:07.298 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2026-02-25T08:47:07.354 INFO:tasks.rados.rados.0.trial169.stdout:1244: finishing write tid 1 to trial16941936-46 2026-02-25T08:47:07.355 INFO:tasks.rados.rados.0.trial169.stdout:1244: finishing write tid 2 to trial16941936-46 2026-02-25T08:47:07.355 INFO:tasks.rados.rados.0.trial169.stdout:1242: done (3 left) 2026-02-25T08:47:07.355 INFO:tasks.rados.rados.0.trial169.stdout:1248: done (2 left) 2026-02-25T08:47:07.355 INFO:tasks.rados.rados.0.trial169.stdout:1249: snap_remove snap 143 2026-02-25T08:47:08.172 INFO:tasks.rados.rados.0.trial169.stdout:1249: done (2 left) 2026-02-25T08:47:08.172 INFO:tasks.rados.rados.0.trial169.stdout:1250: write (excl) oid 12 current snap is 157 2026-02-25T08:47:08.172 INFO:tasks.rados.rados.0.trial169.stdout:1244: finishing write tid 3 to trial16941936-46 2026-02-25T08:47:08.172 INFO:tasks.rados.rados.0.trial169.stdout:1250: seq_num 594 ranges {783358=573440,2026304=580868,3279140=621458} 2026-02-25T08:47:08.177 INFO:tasks.rados.rados.0.trial169.stdout:1250: writing trial16941936-12 from 783358 to 1356798 tid 1 2026-02-25T08:47:08.182 INFO:tasks.rados.rados.0.trial169.stdout:1250: writing trial16941936-12 from 2026304 to 2607172 tid 2 2026-02-25T08:47:08.187 INFO:tasks.rados.rados.0.trial169.stdout:1250: writing trial16941936-12 from 3279140 to 3900598 tid 3 2026-02-25T08:47:08.187 INFO:tasks.rados.rados.0.trial169.stdout:1251: write (excl) oid 22 current snap is 157 2026-02-25T08:47:08.188 INFO:tasks.rados.rados.0.trial169.stdout:1244: finishing write tid 4 to trial16941936-46 2026-02-25T08:47:08.188 INFO:tasks.rados.rados.0.trial169.stdout:1251: seq_num 595 ranges {400165=648689,1684786=315379} 2026-02-25T08:47:08.191 INFO:tasks.rados.rados.0.trial169.stdout:1251: writing trial16941936-22 from 400165 to 1048854 tid 1 2026-02-25T08:47:08.195 INFO:tasks.rados.rados.0.trial169.stdout:1251: writing trial16941936-22 from 1684786 to 2000165 tid 2 2026-02-25T08:47:08.196 INFO:tasks.rados.rados.0.trial169.stdout:1252: read oid 30 snap -1 2026-02-25T08:47:08.196 INFO:tasks.rados.rados.0.trial169.stdout:1252: expect (ObjNum 586 snap 156 seq_num 586) 2026-02-25T08:47:08.196 INFO:tasks.rados.rados.0.trial169.stdout:1253: snap_create 2026-02-25T08:47:08.196 INFO:tasks.rados.rados.0.trial169.stdout:1244: finishing write tid 5 to trial16941936-46 2026-02-25T08:47:08.196 INFO:tasks.rados.rados.0.trial169.stdout:1244: finishing write tid 6 to trial16941936-46 2026-02-25T08:47:08.196 INFO:tasks.rados.rados.0.trial169.stdout:1244: oid 46 updating version 0 to 208 2026-02-25T08:47:08.196 INFO:tasks.rados.rados.0.trial169.stdout:1244: oid 46 version 208 is already newer than 208 2026-02-25T08:47:08.196 INFO:tasks.rados.rados.0.trial169.stdout:1244: oid 46 version 208 is already newer than 208 2026-02-25T08:47:08.196 INFO:tasks.rados.rados.0.trial169.stdout:1244: oid 46 version 208 is already newer than 208 2026-02-25T08:47:08.196 INFO:tasks.rados.rados.0.trial169.stdout:1244: oid 46 version 208 is already newer than 208 2026-02-25T08:47:08.196 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 46 v 208 (ObjNum 590 snap 157 seq_num 590) dirty exists 2026-02-25T08:47:08.197 INFO:tasks.rados.rados.0.trial169.stdout:1244: left oid 46 (ObjNum 590 snap 157 seq_num 590) 2026-02-25T08:47:08.197 INFO:tasks.rados.rados.0.trial169.stdout:1245: finishing write tid 1 to trial16941936-45 2026-02-25T08:47:08.197 INFO:tasks.rados.rados.0.trial169.stdout:1245: finishing write tid 2 to trial16941936-45 2026-02-25T08:47:08.197 INFO:tasks.rados.rados.0.trial169.stdout:1245: finishing write tid 3 to trial16941936-45 2026-02-25T08:47:08.197 INFO:tasks.rados.rados.0.trial169.stdout:1245: finishing write tid 4 to trial16941936-45 2026-02-25T08:47:08.197 INFO:tasks.rados.rados.0.trial169.stdout:1245: finishing write tid 5 to trial16941936-45 2026-02-25T08:47:08.197 INFO:tasks.rados.rados.0.trial169.stdout:1245: finishing write tid 6 to trial16941936-45 2026-02-25T08:47:08.197 INFO:tasks.rados.rados.0.trial169.stdout:1245: oid 45 updating version 0 to 213 2026-02-25T08:47:08.197 INFO:tasks.rados.rados.0.trial169.stdout:1245: oid 45 version 213 is already newer than 213 2026-02-25T08:47:08.197 INFO:tasks.rados.rados.0.trial169.stdout:1245: oid 45 version 213 is already newer than 213 2026-02-25T08:47:08.197 INFO:tasks.rados.rados.0.trial169.stdout:1245: oid 45 version 213 is already newer than 213 2026-02-25T08:47:08.197 INFO:tasks.rados.rados.0.trial169.stdout:1245: oid 45 version 213 is already newer than 213 2026-02-25T08:47:08.198 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 45 v 213 (ObjNum 591 snap 157 seq_num 591) dirty exists 2026-02-25T08:47:08.198 INFO:tasks.rados.rados.0.trial169.stdout:1245: left oid 45 (ObjNum 591 snap 157 seq_num 591) 2026-02-25T08:47:08.209 INFO:tasks.rados.rados.0.trial169.stdout:1252: expect (ObjNum 586 snap 156 seq_num 586) 2026-02-25T08:47:08.217 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 3022819 2026-02-25T08:47:08.239 INFO:tasks.rados.rados.0.trial169.stdout:seeking to 3201193 2026-02-25T08:47:08.259 INFO:tasks.rados.rados.0.trial169.stdout:1250: finishing write tid 1 to trial16941936-12 2026-02-25T08:47:08.259 INFO:tasks.rados.rados.0.trial169.stdout:1251: finishing write tid 1 to trial16941936-22 2026-02-25T08:47:08.259 INFO:tasks.rados.rados.0.trial169.stdout:1250: finishing write tid 2 to trial16941936-12 2026-02-25T08:47:08.260 INFO:tasks.rados.rados.0.trial169.stdout:1250: finishing write tid 3 to trial16941936-12 2026-02-25T08:47:08.260 INFO:tasks.rados.rados.0.trial169.stdout:1251: finishing write tid 2 to trial16941936-22 2026-02-25T08:47:08.260 INFO:tasks.rados.rados.0.trial169.stdout:1250: finishing write tid 4 to trial16941936-12 2026-02-25T08:47:08.260 INFO:tasks.rados.rados.0.trial169.stdout:1250: finishing write tid 5 to trial16941936-12 2026-02-25T08:47:08.260 INFO:tasks.rados.rados.0.trial169.stdout:1250: oid 12 updating version 0 to 90 2026-02-25T08:47:08.260 INFO:tasks.rados.rados.0.trial169.stdout:1250: oid 12 updating version 90 to 93 2026-02-25T08:47:08.260 INFO:tasks.rados.rados.0.trial169.stdout:1250: oid 12 version 93 is already newer than 93 2026-02-25T08:47:08.260 INFO:tasks.rados.rados.0.trial169.stdout:1250: oid 12 version 93 is already newer than 93 2026-02-25T08:47:08.260 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 12 v 93 (ObjNum 594 snap 157 seq_num 594) dirty exists 2026-02-25T08:47:08.260 INFO:tasks.rados.rados.0.trial169.stdout:1250: left oid 12 (ObjNum 594 snap 157 seq_num 594) 2026-02-25T08:47:08.260 INFO:tasks.rados.rados.0.trial169.stdout:1251: finishing write tid 3 to trial16941936-22 2026-02-25T08:47:08.261 INFO:tasks.rados.rados.0.trial169.stdout:1251: finishing write tid 4 to trial16941936-22 2026-02-25T08:47:08.261 INFO:tasks.rados.rados.0.trial169.stdout:1251: oid 22 updating version 0 to 245 2026-02-25T08:47:08.261 INFO:tasks.rados.rados.0.trial169.stdout:1251: oid 22 version 245 is already newer than 245 2026-02-25T08:47:08.261 INFO:tasks.rados.rados.0.trial169.stdout:1251: oid 22 version 245 is already newer than 243 2026-02-25T08:47:08.261 INFO:tasks.rados.rados.0.trial169.stdout:update_object_version oid 22 v 245 (ObjNum 595 snap 157 seq_num 595) dirty exists 2026-02-25T08:47:08.261 INFO:tasks.rados.rados.0.trial169.stdout:1251: left oid 22 (ObjNum 595 snap 157 seq_num 595) 2026-02-25T08:47:08.725 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~303s 2026-02-25T08:47:08.725 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2026-02-25T08:47:09.172 INFO:tasks.rados.rados.0.trial169.stdout:1244: done (5 left) 2026-02-25T08:47:09.172 INFO:tasks.rados.rados.0.trial169.stdout:1245: done (4 left) 2026-02-25T08:47:09.172 INFO:tasks.rados.rados.0.trial169.stdout:1250: done (3 left) 2026-02-25T08:47:09.172 INFO:tasks.rados.rados.0.trial169.stdout:1251: done (2 left) 2026-02-25T08:47:09.172 INFO:tasks.rados.rados.0.trial169.stdout:1252: done (1 left) 2026-02-25T08:47:09.172 INFO:tasks.rados.rados.0.trial169.stdout:1253: done (0 left) 2026-02-25T08:47:09.173 INFO:tasks.rados.rados.0.trial169.stdout:1254: snap_create 2026-02-25T08:47:09.229 INFO:tasks.ceph.osd.1:Sent signal 15 2026-02-25T08:47:09.230 INFO:tasks.ceph.osd.2:Sent signal 15 2026-02-25T08:47:09.230 INFO:tasks.ceph.osd.3:Sent signal 15 2026-02-25T08:47:09.231 INFO:tasks.ceph.mon.a:Sent signal 15 2026-02-25T08:47:09.231 INFO:tasks.ceph.mgr.x:Sent signal 15 2026-02-25T08:47:09.231 INFO:tasks.daemonwatchdog.daemon_watchdog:Killing thrasher OSDThrasher 2026-02-25T08:47:09.231 INFO:tasks.ceph.mon.a.trial169.stderr:2026-02-25T08:47:09.229+0000 7f3f00ffc640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mon -f --cluster ceph -i a (PID: 39820) UID: 0 2026-02-25T08:47:09.231 INFO:tasks.ceph.mon.a.trial169.stderr:2026-02-25T08:47:09.229+0000 7f3f00ffc640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2026-02-25T08:47:09.243 INFO:tasks.ceph.mgr.x.trial183.stderr:2026-02-25T08:47:09.242+0000 7f3043df8640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper kill ceph-mgr -f --cluster ceph -i x (PID: 39370) UID: 0 2026-02-25T08:47:09.244 INFO:tasks.ceph.mgr.x.trial183.stderr:2026-02-25T08:47:09.242+0000 7f3043df8640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2026-02-25T08:47:12.000 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:47:12.071 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:12.074 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:12.075 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:47:17.076 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:47:17.149 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:17.151 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:17.152 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:47:22.153 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:47:22.225 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:22.228 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:47:27.230 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:47:27.301 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:27.305 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:47:32.306 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:47:32.378 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:32.381 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:32.381 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:47:37.382 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:47:37.456 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:37.459 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:37.459 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:47:42.461 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:47:42.535 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:42.538 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:47:47.539 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:47:47.610 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:47.613 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:47.613 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:47:52.615 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:47:52.687 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:52.691 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:47:57.692 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:47:57.764 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:47:57.767 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:47:57.768 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-25T08:48:02.769 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-25T08:48:02.842 INFO:teuthology.orchestra.run.trial169.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2026-02-25T08:48:02.844 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-25T08:48:02.849 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 192, in wrapper return func(self) ^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1513, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 562, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 3236, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2085, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.0 restart 2026-02-25T08:48:02.849 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1384, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 192, in wrapper return func(self) ^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1513, in _do_thrash self.choose_action()() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 562, in revive_osd self.ceph_manager.revive_osd( File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 3236, in revive_osd self.wait_run_admin_socket('osd', osd, File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2085, in wait_run_admin_socket raise Exception('timed out waiting for admin_socket ' Exception: timed out waiting for admin_socket to appear after osd.0 restart 2026-02-25T08:48:02.851 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2026-02-25T08:48:02.851 INFO:tasks.daemonwatchdog.daemon_watchdog:Killing remote process ceph-test-rados-ceph 2026-02-25T08:48:02.851 DEBUG:tasks.rados:Stopping ceph-test-rados-ceph due to exception The WatchDog barked due to Daemon ceph.osd.0 has failed 2026-02-25T08:48:02.851 INFO:tasks.rados:Stopping instance 0 2026-02-25T08:48:02.859 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.0 has failed 2026-02-25T08:48:02.861 ERROR:teuthology.run_tasks:Manager failed: rados Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.0 has failed 2026-02-25T08:48:02.863 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2026-02-25T08:48:02.879 INFO:tasks.thrashosds:joining thrashosds 2026-02-25T08:48:02.879 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2026-02-25T08:48:02.880 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-25T08:48:02.880 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-25T08:48:02.975 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:48:02.971+0000 7f988959c640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:50:02.930 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-25T08:50:02.931 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.0 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2970, in wait_for_all_osds_up while not self.are_all_osds_up(): ^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2960, in are_all_osds_up x = self.get_osd_dump() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2670, in get_osd_dump return self.get_osd_dump_json()['osds'] ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2662, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial169 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json' 2026-02-25T08:50:02.938 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-25T08:50:02.960 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-25T08:50:02.960 DEBUG:teuthology.orchestra.run.trial169:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-25T08:50:03.039 INFO:teuthology.orchestra.run.trial169.stderr:2026-02-25T08:50:03.035+0000 7fef0c30b640 -1 WARNING: the following dangerous and experimental features are enabled: crimson 2026-02-25T08:52:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-25T08:52:02.995 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.0 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial169 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2026-02-25T08:52:02.999 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-25T08:52:02.999 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-25T08:52:02.999 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-25T08:52:03.000 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:52:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:52:03.000 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.0 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 1525, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial169 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial169 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2>> /var/log/ceph/ceph-osd.0.log' 2026-02-25T08:52:03.005 INFO:tasks.ceph.osd.0:Stopped 2026-02-25T08:52:03.005 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-25T08:52:03.006 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:52:03.078 INFO:tasks.ceph.osd.1:Stopped 2026-02-25T08:52:03.079 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-25T08:52:03.079 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:52:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:52:03.079 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.0 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 1525, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial169 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial183 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2>> /var/log/ceph/ceph-osd.2.log' 2026-02-25T08:52:03.085 INFO:tasks.ceph.osd.2:Stopped 2026-02-25T08:52:03.085 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-25T08:52:03.085 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:52:03.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:52:03.086 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.0 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 1525, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial169 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/daemon/state.py", line 146, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial183 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2>> /var/log/ceph/ceph-osd.3.log' 2026-02-25T08:52:03.091 INFO:tasks.ceph.osd.3:Stopped 2026-02-25T08:52:03.091 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-25T08:52:03.092 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-25T08:52:03.092 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:52:03.092 INFO:tasks.ceph.mgr.x:Stopped 2026-02-25T08:52:03.092 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-25T08:52:03.092 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-25T08:52:03.092 INFO:teuthology.orchestra.run:waiting for 300 2026-02-25T08:52:03.093 INFO:tasks.ceph.mon.a:Stopped 2026-02-25T08:52:03.093 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-25T08:52:03.093 DEBUG:teuthology.orchestra.run.trial169:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v '\(POOL_APP_NOT_ENABLED\)' | grep -E -v 'but it is still running' | grep -E -v 'objects unfound and apparently lost' | grep -E -v 'overall HEALTH_' | grep -E -v '\(OSDMAP_FLAGS\)' | grep -E -v '\(OSD_' | grep -E -v '\(PG_' | grep -E -v '\(POOL_' | grep -E -v '\(CACHE_POOL_' | grep -E -v '\(SMALLER_PGP_NUM\)' | grep -E -v '\(OBJECT_' | grep -E -v SLOW_OPS | grep -E -v '\(REQUEST_SLOW\)' | grep -E -v '\(TOO_FEW_PGS\)' | grep -E -v 'slow request' | grep -E -v 'timeout on replica' | grep -E -v 'late reservation from' | grep -E -v MON_DOWN | grep -E -v OSDMAP_FLAGS | grep -E -v OSD_DOWN | grep -E -v PG_DEGRADED | grep -E -v PG_AVAILABILITY | grep -E -v POOL_APP_NOT_ENABLED | grep -E -v 'mons down' | grep -E -v 'mon down' | grep -E -v 'out of quorum' | grep -E -v noscrub | grep -E -v nodeep-scrub | grep -E -v 'Degraded data redundancy' | grep -E -v 'is down' | grep -E -v 'osds down' | grep -E -v 'pg .*? is .*?degraded.*?, acting' | grep -E -v 'pg .*? is stuck' | grep -E -v 'pg degraded' | grep -E -v PG_BACKFILL_FULL | grep -E -v 'Low space hindering backfill .*? backfill_toofull' | grep -E -v OSD_ROOT_DOWN | grep -E -v 'pgs degraded' | grep -E -v 'pgs undersized' | grep -E -v 'is active.*backfill_toofull.*' | head -n 1 2026-02-25T08:52:03.120 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial169.front.sepia.ceph.com 2026-02-25T08:52:03.120 DEBUG:teuthology.orchestra.run.trial169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-25T08:52:03.186 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial169.front.sepia.ceph.com 2026-02-25T08:52:03.186 DEBUG:teuthology.orchestra.run.trial169:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-25T08:52:03.258 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial183.front.sepia.ceph.com 2026-02-25T08:52:03.258 DEBUG:teuthology.orchestra.run.trial183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-25T08:52:03.300 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@trial183.front.sepia.ceph.com 2026-02-25T08:52:03.300 DEBUG:teuthology.orchestra.run.trial183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2026-02-25T08:52:03.367 INFO:tasks.ceph:Archiving mon data... 2026-02-25T08:52:03.369 DEBUG:teuthology.misc:Transferring archived files from trial169:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69078/data/mon.a.tgz 2026-02-25T08:52:03.369 DEBUG:teuthology.orchestra.run.trial169:> mktemp 2026-02-25T08:52:03.387 INFO:teuthology.orchestra.run.trial169.stdout:/tmp/tmp.qifbLNkAm6 2026-02-25T08:52:03.388 DEBUG:teuthology.orchestra.run.trial169:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.qifbLNkAm6 2026-02-25T08:52:03.499 DEBUG:teuthology.orchestra.run.trial169:> sudo chmod 0666 /tmp/tmp.qifbLNkAm6 2026-02-25T08:52:03.579 DEBUG:teuthology.orchestra.remote:trial169:/tmp/tmp.qifbLNkAm6 is 406KB 2026-02-25T08:52:03.671 DEBUG:teuthology.orchestra.run.trial169:> rm -fr /tmp/tmp.qifbLNkAm6 2026-02-25T08:52:03.688 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-25T08:52:03.688 DEBUG:teuthology.orchestra.run.trial169:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-25T08:52:03.729 DEBUG:teuthology.orchestra.run.trial183:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-25T08:52:03.808 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-25T08:52:03.825 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-25T08:52:03.825 INFO:tasks.ceph:Archiving crash dumps... 2026-02-25T08:52:03.828 DEBUG:teuthology.misc:Transferring archived files from trial169:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69078/remote/trial169/crash 2026-02-25T08:52:03.829 DEBUG:teuthology.orchestra.run.trial169:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-25T08:52:03.875 DEBUG:teuthology.misc:Transferring archived files from trial183:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69078/remote/trial183/crash 2026-02-25T08:52:03.876 DEBUG:teuthology.orchestra.run.trial183:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-25T08:52:03.903 INFO:tasks.ceph:Compressing logs... 2026-02-25T08:52:03.903 DEBUG:teuthology.orchestra.run.trial169:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:52:03.919 DEBUG:teuthology.orchestra.run.trial183:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:52:03.941 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46815.log 2026-02-25T08:52:03.941 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41106.log 2026-02-25T08:52:03.941 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46815.log: gzip -5 50.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.46815.log.gz -- 2026-02-25T08:52:03.942 INFO:teuthology.orchestra.run.trial169.stderr: /var/log/ceph/ceph-client.admin.51138.log 2026-02-25T08:52:03.942 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48976.log 2026-02-25T08:52:03.942 INFO:teuthology.orchestra.run.trial169.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.41106.log.gz 2026-02-25T08:52:03.945 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51138.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51138.log.gz 2026-02-25T08:52:03.946 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45437.log 2026-02-25T08:52:03.946 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51192.log 2026-02-25T08:52:03.946 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48976.log.gz 2026-02-25T08:52:03.946 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45437.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-25T08:52:03.946 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45437.log.gz 2026-02-25T08:52:03.946 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51192.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51192.log.gz 2026-02-25T08:52:03.946 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41813.log 2026-02-25T08:52:03.946 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48004.log 2026-02-25T08:52:03.946 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43734.log 2026-02-25T08:52:03.946 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41813.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.41813.log.gz 2026-02-25T08:52:03.946 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42987.log 2026-02-25T08:52:03.946 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48004.log.gz 2026-02-25T08:52:03.946 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.43734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49949.log 2026-02-25T08:52:03.946 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43734.log.gz 2026-02-25T08:52:03.947 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49246.log 2026-02-25T08:52:03.947 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42987.log.gz 2026-02-25T08:52:03.947 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49949.log: gzip 50.0% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.49949.log.gz /var/log/ceph/ceph-client.admin.51707.log 2026-02-25T08:52:03.947 INFO:teuthology.orchestra.run.trial169.stderr: 2026-02-25T08:52:03.947 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45383.log 2026-02-25T08:52:03.947 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49246.log.gz 2026-02-25T08:52:03.947 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51707.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-25T08:52:03.947 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51707.log.gz 2026-02-25T08:52:03.947 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45383.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45383.log.gz 2026-02-25T08:52:03.947 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41505.log 2026-02-25T08:52:03.947 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40839.log 2026-02-25T08:52:03.947 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.41505.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41505.log.gz 2026-02-25T08:52:03.947 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45518.log 2026-02-25T08:52:03.957 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47464.log 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40839.log.gz 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45518.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45518.log.gz 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50868.log 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41057.log 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47464.log.gz 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50868.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50868.log.gz 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50544.log 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51057.log 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41057.log.gz 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50544.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50544.log.gz 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45302.log 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43815.log 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51057.log.gz 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45302.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45302.log.gz 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41503.log 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.43815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40518.log 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43815.log.gz 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41503.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41503.log.gz 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50679.log 2026-02-25T08:52:03.958 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40518.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40518.log.gz 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52463.log 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50679.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50679.log.gz 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51517.log 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.52463.log: 8.8% -- replaced with /var/log/ceph/ceph-client.admin.52463.log.gz 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46085.log 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51517.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.51517.log.gz -5 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42628.log 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46085.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46085.log.gz 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40893.log 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42628.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42628.log.gz -5 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50273.log 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40893.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40893.log.gz 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47599.log 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50273.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.50273.log.gz 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44519.log 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47599.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47599.log.gz 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45842.log 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44519.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44519.log.gz 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48517.log 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45842.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.45842.log.gz 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40694.log 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48517.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48517.log.gz 2026-02-25T08:52:03.959 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45059.log 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40694.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40694.log.gz 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.41936.log 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45059.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45059.log.gz 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45707.log 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.0.41936.log: 50.0% -- replaced with /var/log/ceph/ceph-client.0.41936.log.gz 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48328.log 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45707.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45707.log.gz 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46950.log 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48328.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.48328.log.gz 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47004.log 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46950.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46950.log.gz 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47923.log 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42207.log 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47004.log.gz 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47923.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47923.log.gz 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40838.log 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42685.log 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42207.log.gz 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40838.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40838.log.gz 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52059.log 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42685.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42685.log.gz 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51652.log 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.52059.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.52059.log.gz 2026-02-25T08:52:03.960 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48571.log 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49814.log 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51652.log.gz 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48571.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48571.log.gz 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48193.log 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49814.log: 49.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.49814.log.gz 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42475.log 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48193.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48193.log.gz 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41504.log 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42475.log: gzip -5 --verbose -- 50.4% /var/log/ceph/ceph-client.admin.41378.log 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42475.log.gz 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41504.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41504.log.gz 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45113.log 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41378.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41378.log.gz 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44194.log 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45113.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45113.log.gz 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46680.log 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44194.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44194.log.gz 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47545.log 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46680.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46680.log.gz 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40696.log 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47545.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47545.log.gz 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47869.log 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40696.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40696.log.gz 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47085.log 2026-02-25T08:52:03.961 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47869.log: 50.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.47869.log.gz 2026-02-25T08:52:03.962 INFO:teuthology.orchestra.run.trial169.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40320.log 2026-02-25T08:52:03.962 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47085.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47085.log.gz 2026-02-25T08:52:03.962 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45761.log 2026-02-25T08:52:03.962 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40320.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.40320.log.gz --verbose 2026-02-25T08:52:03.962 INFO:teuthology.orchestra.run.trial169.stderr: -- /var/log/ceph/ceph-client.admin.41302.log 2026-02-25T08:52:03.966 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45761.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45761.log.gz 2026-02-25T08:52:03.966 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49489.log 2026-02-25T08:52:03.966 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41302.log: gzip 49.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.41302.log.gz --verbose 2026-02-25T08:52:03.966 INFO:teuthology.orchestra.run.trial169.stderr: -- /var/log/ceph/ceph-client.admin.43599.log 2026-02-25T08:52:03.966 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49489.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49489.log.gz 2026-02-25T08:52:03.966 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51842.log 2026-02-25T08:52:03.966 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.43599.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43599.log.gz 2026-02-25T08:52:03.966 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41706.log 2026-02-25T08:52:03.966 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44924.log 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.51842.log.gz 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41706.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41706.log.gz 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45896.log 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42937.log 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44924.log.gz 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41155.log 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45896.log.gz 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42937.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42937.log.gz 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44329.log 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50733.log 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41155.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41155.log.gz 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44329.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44329.log.gz 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40420.log 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50408.log 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50733.log.gz 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40420.log: 50.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.40420.log.gz 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48787.log 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45248.log 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.50408.log.gz 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50084.log 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48787.log.gz 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45248.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45248.log.gz 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47653.log 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50084.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50084.log.gz 2026-02-25T08:52:03.967 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45977.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47653.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.38759.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47653.log.gz 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45977.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.45977.log.gz 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48436.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph.tmp-client.admin.38759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42810.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr: 67.7% -- replaced with /var/log/ceph/ceph.tmp-client.admin.38759.log.gz 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48436.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48436.log.gz 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43355.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41734.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42810.log.gz 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.43355.log: gzip 50.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.43355.log.gz --verbose 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr: -- /var/log/ceph/ceph-client.admin.45194.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43869.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41734.log.gz 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45194.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.45194.log.gz 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49760.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.43869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43409.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43869.log.gz 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46545.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49760.log.gz 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.43409.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43409.log.gz 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49895.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42051.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46545.log.gz 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49895.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49895.log.gz 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40469.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42051.log: gzip -5 --verbose -- 50.0% /var/log/ceph/ceph-client.admin.47274.log 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42051.log.gz 2026-02-25T08:52:03.968 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48841.log 2026-02-25T08:52:03.969 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40469.log.gz 2026-02-25T08:52:03.971 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47274.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47274.log.gz 2026-02-25T08:52:03.971 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50814.log 2026-02-25T08:52:03.971 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51273.log 2026-02-25T08:52:03.971 INFO:teuthology.orchestra.run.trial169.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48841.log.gz 2026-02-25T08:52:03.971 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50814.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50814.log.gz 2026-02-25T08:52:03.971 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40695.log 2026-02-25T08:52:03.971 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42888.log 2026-02-25T08:52:03.971 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51273.log.gz 2026-02-25T08:52:03.971 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40695.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40695.log.gz 2026-02-25T08:52:03.971 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49030.log 2026-02-25T08:52:03.971 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47734.log 2026-02-25T08:52:03.971 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42888.log.gz 2026-02-25T08:52:03.971 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49030.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49030.log.gz 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51327.log 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44275.log 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47734.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47734.log.gz 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51327.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51327.log.gz 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43951.log 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42363.log 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44275.log.gz 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.43951.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43951.log.gz 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40347.log 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42363.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42363.log.gz 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46761.log 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42265.log 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr: 66.9% -- replaced with /var/log/ceph/ceph-client.admin.40347.log.gz 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46761.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46761.log.gz 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44978.log 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42531.log 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42265.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42265.log.gz 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44978.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44978.log.gz 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50030.log 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41635.log 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42531.log.gz 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50030.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50030.log.gz 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39823.log 2026-02-25T08:52:03.972 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39909.log 2026-02-25T08:52:03.974 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-25T08:52:03.974 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2026-02-25T08:52:03.974 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39568.log 2026-02-25T08:52:03.974 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39470.log 2026-02-25T08:52:03.974 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39622.log 2026-02-25T08:52:03.974 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/ceph-client.admin.39568.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39568.log.gz 2026-02-25T08:52:03.974 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/ceph-client.admin.39470.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-25T08:52:03.974 INFO:teuthology.orchestra.run.trial183.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39470.log.gz 2026-02-25T08:52:03.974 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/ceph-client.admin.39622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39517.log 2026-02-25T08:52:03.974 INFO:teuthology.orchestra.run.trial183.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39622.log.gz 2026-02-25T08:52:03.974 INFO:teuthology.orchestra.run.trial183.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.39517.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39517.log.gz 2026-02-25T08:52:03.975 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41635.log.gz 2026-02-25T08:52:03.975 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.39823.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.39823.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51003.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.39909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44654.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39909.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51003.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51003.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52425.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48112.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44654.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.52425.log: 8.0% -- replaced with /var/log/ceph/ceph-client.admin.52425.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50219.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.48112.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51923.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43545.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr: 87.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50219.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51923.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51923.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47220.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.43545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48382.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43545.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47220.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47220.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42106.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45626.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48382.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42106.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42106.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51382.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43043.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45626.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51382.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.51382.log.gz 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49679.log 2026-02-25T08:52:03.976 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.43043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43099.log 2026-02-25T08:52:03.977 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43043.log.gz 2026-02-25T08:52:03.977 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49679.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49679.log.gz 2026-02-25T08:52:03.977 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-25T08:52:03.977 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.43099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46869.log 2026-02-25T08:52:03.977 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43099.log.gz 2026-02-25T08:52:03.978 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41008.log 2026-02-25T08:52:03.978 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44140.log 2026-02-25T08:52:03.978 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46869.log.gz 2026-02-25T08:52:03.978 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40567.log 2026-02-25T08:52:03.978 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41008.log.gz 2026-02-25T08:52:03.978 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44140.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44140.log.gz 2026-02-25T08:52:03.978 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50354.log 2026-02-25T08:52:03.978 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44411.log 2026-02-25T08:52:03.978 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40567.log.gz 2026-02-25T08:52:03.978 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50354.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50354.log.gz 2026-02-25T08:52:03.978 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46356.log 2026-02-25T08:52:03.978 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44005.log 2026-02-25T08:52:03.978 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44411.log.gz 2026-02-25T08:52:03.979 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43490.log 2026-02-25T08:52:03.979 INFO:teuthology.orchestra.run.trial169.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.46356.log.gz 2026-02-25T08:52:03.979 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49165.log 2026-02-25T08:52:03.979 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44005.log.gz 2026-02-25T08:52:03.979 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.43490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49111.log 2026-02-25T08:52:03.979 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.43490.log.gz 2026-02-25T08:52:03.979 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48247.log 2026-02-25T08:52:03.980 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49165.log.gz 2026-02-25T08:52:03.980 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40190.log 2026-02-25T08:52:03.980 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49111.log.gz 2026-02-25T08:52:03.980 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43680.log 2026-02-25T08:52:03.980 INFO:teuthology.orchestra.run.trial169.stderr: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.48247.log.gz 2026-02-25T08:52:03.981 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40892.log 2026-02-25T08:52:03.981 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.40190.log.gz 2026-02-25T08:52:03.981 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51977.log 2026-02-25T08:52:03.981 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.43680.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.43680.log.gz 2026-02-25T08:52:03.981 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51463.log 2026-02-25T08:52:03.981 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40892.log.gz 2026-02-25T08:52:03.981 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46302.log 2026-02-25T08:52:03.981 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51977.log.gz 2026-02-25T08:52:03.981 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50922.log 2026-02-25T08:52:03.982 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51463.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51463.log.gz 2026-02-25T08:52:03.982 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46302.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46302.log.gz 2026-02-25T08:52:03.982 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40250.log 2026-02-25T08:52:03.982 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41935.log 2026-02-25T08:52:03.982 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50922.log.gz 2026-02-25T08:52:03.983 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40164.log 2026-02-25T08:52:03.983 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40250.log.gz 2026-02-25T08:52:03.983 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51598.log 2026-02-25T08:52:03.983 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41935.log.gz 2026-02-25T08:52:03.983 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40164.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40164.log.gz 2026-02-25T08:52:03.983 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46412.log 2026-02-25T08:52:03.983 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49624.log 2026-02-25T08:52:03.983 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51598.log.gz 2026-02-25T08:52:03.983 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46412.log: gzip -5 50.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.46412.log.gz -- 2026-02-25T08:52:03.983 INFO:teuthology.orchestra.run.trial169.stderr: /var/log/ceph/ceph-client.admin.44735.log 2026-02-25T08:52:03.984 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46167.log 2026-02-25T08:52:03.984 INFO:teuthology.orchestra.run.trial169.stderr: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.49624.log.gz 2026-02-25T08:52:03.984 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46221.log 2026-02-25T08:52:03.984 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44735.log.gz 2026-02-25T08:52:03.984 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47788.log 2026-02-25T08:52:03.984 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46167.log.gz 2026-02-25T08:52:03.984 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41204.log 2026-02-25T08:52:03.985 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46221.log.gz 2026-02-25T08:52:03.985 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47329.log 2026-02-25T08:52:03.985 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47788.log.gz 2026-02-25T08:52:03.985 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44600.log 2026-02-25T08:52:03.985 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.41204.log.gz 2026-02-25T08:52:03.985 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47139.log 2026-02-25T08:52:03.985 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47329.log.gz 2026-02-25T08:52:03.985 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44600.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44600.log.gz 2026-02-25T08:52:03.985 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44843.log 2026-02-25T08:52:03.986 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45572.log 2026-02-25T08:52:03.986 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.47139.log.gz 2026-02-25T08:52:03.986 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44843.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44843.log.gz 2026-02-25T08:52:03.986 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44465.log 2026-02-25T08:52:03.986 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.45572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50138.log 2026-02-25T08:52:03.987 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.45572.log.gz 2026-02-25T08:52:03.987 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50463.log 2026-02-25T08:52:03.987 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44465.log.gz 2026-02-25T08:52:03.987 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50138.log: gzip -5 --verbose 50.0% -- -- replaced with /var/log/ceph/ceph-client.admin.50138.log.gz /var/log/ceph/ceph-client.admin.46031.log 2026-02-25T08:52:03.987 INFO:teuthology.orchestra.run.trial169.stderr: 2026-02-25T08:52:03.987 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49300.log 2026-02-25T08:52:03.987 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.50463.log.gz 2026-02-25T08:52:03.987 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46031.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.46031.log.gz 2026-02-25T08:52:03.987 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47410.log 2026-02-25T08:52:03.988 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42314.log 2026-02-25T08:52:03.988 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49300.log.gz 2026-02-25T08:52:03.988 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.47410.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.47410.log.gz 2026-02-25T08:52:03.988 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49354.log 2026-02-25T08:52:03.988 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42314.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.42314.log.gz 2026-02-25T08:52:03.988 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44059.log 2026-02-25T08:52:03.988 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49354.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49354.log.gz 2026-02-25T08:52:03.988 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-25T08:52:03.989 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44059.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.44059.log.gz 2026-02-25T08:52:03.989 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42001.log 2026-02-25T08:52:03.989 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42809.log 2026-02-25T08:52:03.989 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42001.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42001.log.gz 2026-02-25T08:52:03.989 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40697.log 2026-02-25T08:52:03.989 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41634.log 2026-02-25T08:52:03.989 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.42809.log.gz 2026-02-25T08:52:03.989 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.40697.log: 87.7% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-25T08:52:03.990 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.40697.log.gz 2026-02-25T08:52:03.990 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42419.log 2026-02-25T08:52:03.990 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42155.log 2026-02-25T08:52:03.990 INFO:teuthology.orchestra.run.trial169.stderr: 48.0% -- replaced with /var/log/ceph/ceph-client.admin.41634.log.gz 2026-02-25T08:52:03.990 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42419.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42419.log.gz 2026-02-25T08:52:03.990 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44789.log 2026-02-25T08:52:03.990 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42155.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42155.log.gz 2026-02-25T08:52:03.990 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46626.log 2026-02-25T08:52:03.991 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.44789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51788.log 2026-02-25T08:52:03.991 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.44789.log.gz 2026-02-25T08:52:03.991 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46626.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.46626.log.gz 2026-02-25T08:52:03.991 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42580.log 2026-02-25T08:52:03.991 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.51788.log: gzip 50.0% -- replaced with /var/log/ceph/ceph-client.admin.51788.log.gz -5 2026-02-25T08:52:03.991 INFO:teuthology.orchestra.run.trial169.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41253.log 2026-02-25T08:52:03.991 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42580.log: 50.4% -- replaced with /var/log/ceph/ceph-client.admin.42580.log.gz 2026-02-25T08:52:03.991 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49435.log 2026-02-25T08:52:03.991 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41253.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41253.log.gz 2026-02-25T08:52:03.991 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41840.log 2026-02-25T08:52:03.992 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49435.log: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.49435.log.gz 2026-02-25T08:52:03.992 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48058.log 2026-02-25T08:52:03.992 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.41840.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.41840.log.gz 2026-02-25T08:52:03.992 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50598.log 2026-02-25T08:52:03.992 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48058.log: 49.2% -- replaced with /var/log/ceph/ceph-client.admin.48058.log.gz 2026-02-25T08:52:03.992 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46491.log 2026-02-25T08:52:03.992 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.50598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48895.log 2026-02-25T08:52:03.992 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.50598.log.gz 2026-02-25T08:52:03.993 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.46491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42734.log 2026-02-25T08:52:03.993 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.46491.log.gz 2026-02-25T08:52:03.993 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49570.log 2026-02-25T08:52:03.993 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48895.log.gz 2026-02-25T08:52:03.993 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.42734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39956.log 2026-02-25T08:52:03.993 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.42734.log.gz 2026-02-25T08:52:03.993 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.49570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48706.log 2026-02-25T08:52:03.993 INFO:teuthology.orchestra.run.trial169.stderr: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.49570.log.gz 2026-02-25T08:52:03.994 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.39956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48652.log 2026-02-25T08:52:03.994 INFO:teuthology.orchestra.run.trial169.stderr: 49.6% -- replaced with /var/log/ceph/ceph-client.admin.39956.log.gz 2026-02-25T08:52:03.994 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48706.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48706.log.gz 2026-02-25T08:52:03.994 INFO:teuthology.orchestra.run.trial169.stderr:/var/log/ceph/ceph-client.admin.48652.log: 50.0% -- replaced with /var/log/ceph/ceph-client.admin.48652.log.gz 2026-02-25T08:52:04.003 INFO:teuthology.orchestra.run.trial183.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-25T08:52:04.072 INFO:teuthology.orchestra.run.trial169.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-25T08:52:04.106 INFO:teuthology.orchestra.run.trial169.stderr: 89.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-25T08:52:05.174 INFO:teuthology.orchestra.run.trial169.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-25T08:52:05.176 INFO:teuthology.orchestra.run.trial169.stderr: 2026-02-25T08:52:05.176 INFO:teuthology.orchestra.run.trial169.stderr:real 0m1.244s 2026-02-25T08:52:05.176 INFO:teuthology.orchestra.run.trial169.stderr:user 0m1.477s 2026-02-25T08:52:05.176 INFO:teuthology.orchestra.run.trial169.stderr:sys 0m0.152s 2026-02-25T08:52:05.499 INFO:teuthology.orchestra.run.trial183.stderr: 92.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2026-02-25T08:52:05.515 INFO:teuthology.orchestra.run.trial183.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-25T08:52:05.517 INFO:teuthology.orchestra.run.trial183.stderr: 2026-02-25T08:52:05.517 INFO:teuthology.orchestra.run.trial183.stderr:real 0m1.554s 2026-02-25T08:52:05.517 INFO:teuthology.orchestra.run.trial183.stderr:user 0m2.940s 2026-02-25T08:52:05.517 INFO:teuthology.orchestra.run.trial183.stderr:sys 0m0.177s 2026-02-25T08:52:05.517 INFO:tasks.ceph:Archiving logs... 2026-02-25T08:52:05.517 DEBUG:teuthology.misc:Transferring archived files from trial169:/var/log/ceph to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69078/remote/trial169/log 2026-02-25T08:52:05.518 DEBUG:teuthology.orchestra.run.trial169:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-25T08:52:05.817 DEBUG:teuthology.misc:Transferring archived files from trial183:/var/log/ceph to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69078/remote/trial183/log 2026-02-25T08:52:05.817 DEBUG:teuthology.orchestra.run.trial183:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-25T08:52:06.111 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.0 has failed During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 158, in __exit__ self.gen.throw(value) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2010, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2936, in wait_for_clean num_active_clean = self.get_num_active_clean() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2715, in get_num_active_clean pgs = self.get_pg_stats() ^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 2481, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1713, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph_manager.py", line 1704, in run_cluster_cmd return self.controller.run(**kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial169 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json' 2026-02-25T08:52:06.112 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T08:52:06.125 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/task/install/__init__.py", line 644, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/ceph.py", line 2000, in task yield File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/thrashosds.py", line 214, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.0 has failed 2026-02-25T08:52:06.243 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-25T08:52:06.255 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/task/install/__init__.py", line 644, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 372, in task running.get() File "src/gevent/greenlet.py", line 831, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/virtualenv/lib/python3.12/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph_c32469671c6b0222f786dd36c34eb702f780a11d/qa/tasks/rados.py", line 352, in thread raise watched_process.exception ^^^^^^^^^^^^^^^^^ tasks.daemonwatchdog.BarkError: The WatchDog barked due to Daemon ceph.osd.0 has failed 2026-02-25T08:52:06.256 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-25T08:52:06.256 DEBUG:teuthology.orchestra.run.trial169:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-25T08:52:06.258 DEBUG:teuthology.orchestra.run.trial183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-25T08:52:06.284 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-25T08:52:06.296 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-25T08:52:06.296 DEBUG:teuthology.orchestra.run.trial169:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:52:06.325 DEBUG:teuthology.orchestra.run.trial183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T08:52:06.338 INFO:teuthology.orchestra.run.trial169.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:52:06.340 INFO:teuthology.orchestra.run.trial183.stderr:bash: line 1: ntpq: command not found 2026-02-25T08:52:06.375 INFO:teuthology.orchestra.run.trial169.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:52:06.375 INFO:teuthology.orchestra.run.trial169.stdout:=============================================================================== 2026-02-25T08:52:06.375 INFO:teuthology.orchestra.run.trial169.stdout:^- time.richie.net 2 8 377 126 -196us[ -196us] +/- 54ms 2026-02-25T08:52:06.375 INFO:teuthology.orchestra.run.trial169.stdout:^- 44.190.5.123 2 7 377 72 +1046us[+1046us] +/- 33ms 2026-02-25T08:52:06.375 INFO:teuthology.orchestra.run.trial169.stdout:^* time.cloudflare.com 3 8 377 203 -201us[ -247us] +/- 6925us 2026-02-25T08:52:06.375 INFO:teuthology.orchestra.run.trial169.stdout:^- us.timeserver.pub 3 7 377 66 +1048us[+1048us] +/- 33ms 2026-02-25T08:52:06.381 INFO:teuthology.orchestra.run.trial183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-25T08:52:06.381 INFO:teuthology.orchestra.run.trial183.stdout:=============================================================================== 2026-02-25T08:52:06.381 INFO:teuthology.orchestra.run.trial183.stdout:^* 107-172-222-7-host.coloc> 2 8 373 133 +128us[ +81us] +/- 13ms 2026-02-25T08:52:06.381 INFO:teuthology.orchestra.run.trial183.stdout:^- ntp.maxhost.io 2 8 377 59 +227us[ +227us] +/- 56ms 2026-02-25T08:52:06.381 INFO:teuthology.orchestra.run.trial183.stdout:^- rabendary.tesp.com 2 8 377 392 +643us[ +624us] +/- 80ms 2026-02-25T08:52:06.381 INFO:teuthology.orchestra.run.trial183.stdout:^+ time.trtnw.net 2 8 377 195 -527us[ -572us] +/- 37ms 2026-02-25T08:52:06.382 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-25T08:52:06.399 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-25T08:52:06.400 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-25T08:52:06.415 DEBUG:teuthology.orchestra.run.trial169:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-25T08:52:06.474 DEBUG:teuthology.orchestra.run.trial183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-25T08:52:06.529 DEBUG:teuthology.orchestra.run.trial169:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:52:06.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:52:06.557 DEBUG:teuthology.orchestra.run.trial183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-25T08:52:06.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:52:06.583 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-25T08:52:06.597 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-25T08:52:06.610 INFO:teuthology.task.internal:Duration was 2757.705515 seconds 2026-02-25T08:52:06.610 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-25T08:52:06.623 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-25T08:52:06.623 DEBUG:teuthology.orchestra.run.trial169:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-25T08:52:06.626 DEBUG:teuthology.orchestra.run.trial183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-25T08:52:06.662 INFO:teuthology.orchestra.run.trial169.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:52:06.663 INFO:teuthology.orchestra.run.trial183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-25T08:52:07.031 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-25T08:52:07.031 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial169.front.sepia.ceph.com 2026-02-25T08:52:07.032 DEBUG:teuthology.orchestra.run.trial169:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-25T08:52:07.090 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial183.front.sepia.ceph.com 2026-02-25T08:52:07.091 DEBUG:teuthology.orchestra.run.trial183:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-25T08:52:07.111 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-25T08:52:07.112 DEBUG:teuthology.orchestra.run.trial169:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:52:07.133 DEBUG:teuthology.orchestra.run.trial183:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:52:07.212 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-25T08:52:07.212 DEBUG:teuthology.orchestra.run.trial169:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:52:07.214 DEBUG:teuthology.orchestra.run.trial183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-25T08:52:07.235 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:52:07.236 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:52:07.236 INFO:teuthology.orchestra.run.trial169.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-25T08:52:07.236 INFO:teuthology.orchestra.run.trial169.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:52:07.236 INFO:teuthology.orchestra.run.trial169.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-25T08:52:07.237 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T08:52:07.237 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T08:52:07.237 INFO:teuthology.orchestra.run.trial183.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-25T08:52:07.237 INFO:teuthology.orchestra.run.trial183.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-25T08:52:07.238 INFO:teuthology.orchestra.run.trial183.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-25T08:52:07.247 INFO:teuthology.orchestra.run.trial169.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 83.8% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-25T08:52:07.248 INFO:teuthology.orchestra.run.trial183.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 82.1% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-25T08:52:07.250 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-25T08:52:07.278 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-25T08:52:07.278 DEBUG:teuthology.orchestra.run.trial169:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-25T08:52:07.313 DEBUG:teuthology.orchestra.run.trial183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-25T08:52:07.336 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-25T08:52:07.347 DEBUG:teuthology.orchestra.run.trial169:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:52:07.354 DEBUG:teuthology.orchestra.run.trial183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:52:07.375 INFO:teuthology.orchestra.run.trial169.stdout:kernel.core_pattern = core 2026-02-25T08:52:07.401 INFO:teuthology.orchestra.run.trial183.stdout:kernel.core_pattern = core 2026-02-25T08:52:07.418 DEBUG:teuthology.orchestra.run.trial169:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:52:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:52:07.444 DEBUG:teuthology.orchestra.run.trial183:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-25T08:52:07.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T08:52:07.474 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-25T08:52:07.496 INFO:teuthology.task.internal:Transferring archived files... 2026-02-25T08:52:07.497 DEBUG:teuthology.misc:Transferring archived files from trial169:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69078/remote/trial169 2026-02-25T08:52:07.497 DEBUG:teuthology.orchestra.run.trial169:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-25T08:52:07.536 DEBUG:teuthology.misc:Transferring archived files from trial183:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/matan-2026-02-25_07:55:35-crimson-rados-wip-logstoremanager-distro-debug-trial/69078/remote/trial183 2026-02-25T08:52:07.537 DEBUG:teuthology.orchestra.run.trial183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-25T08:52:07.571 INFO:teuthology.task.internal:Removing archive directory... 2026-02-25T08:52:07.572 DEBUG:teuthology.orchestra.run.trial169:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-25T08:52:07.574 DEBUG:teuthology.orchestra.run.trial183:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-25T08:52:07.620 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-25T08:52:07.631 INFO:teuthology.task.internal:Not uploading archives. 2026-02-25T08:52:07.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-25T08:52:07.645 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-25T08:52:07.645 DEBUG:teuthology.orchestra.run.trial169:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-25T08:52:07.647 DEBUG:teuthology.orchestra.run.trial183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-25T08:52:07.660 INFO:teuthology.orchestra.run.trial169.stdout: 920834 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 25 08:52 /home/ubuntu/cephtest 2026-02-25T08:52:07.676 INFO:teuthology.orchestra.run.trial183.stdout: 917525 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 25 08:52 /home/ubuntu/cephtest 2026-02-25T08:52:07.677 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-25T08:52:07.688 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-25T08:52:07.705 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: BarkError: The WatchDog barked due to Daemon ceph.osd.0 has failed 2026-02-25T08:52:07.707 INFO:teuthology.run:Summary data: description: crimson-rados/thrash/{0-crimson_install 0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{default} clusters/{crimson-fixed-2} crimson-supported-all-distro/centos_latest crimson_qa_overrides deploy/ceph objectstore/seastore/rbm$/{crimson_seastore_rbm} thrashers/{default} thrashosds-health workloads/snaps-few-objects-balanced} duration: 2757.705514907837 failure_reason: The WatchDog barked due to Daemon ceph.osd.0 has failed flavor: debug owner: scheduled_matan@soko04.front.sepia.ceph.com status: fail success: false 2026-02-25T08:52:07.707 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T08:52:07.810 INFO:teuthology.run:FAIL