2026-02-17T15:08:06.345 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-17T15:08:06.360 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-17T15:08:06.438 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/skanta-2026-02-17_04:10:36-rados-wip-bharath20-testing-2026-02-13-0906-squid-distro-default-trial/51568 branch: wip-bharath20-testing-2026-02-13-0906-squid description: rados/singleton/{all/ec-inconsistent-hinfo mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} email: skanta@redhat.com first_in_suite: false flavor: default job_id: '51568' kernel: branch: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: trial name: skanta-2026-02-17_04:10:36-rados-wip-bharath20-testing-2026-02-13-0906-squid-distro-default-trial no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-bharath20-testing-2026-02-13-0906-squid ceph: conf: global: mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 3c96dfb3f50b3d02598a82af7104764bfdcd3919 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 3c96dfb3f50b3d02598a82af7104764bfdcd3919 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath20-testing-2026-02-13-0906-squid sha1: 3c96dfb3f50b3d02598a82af7104764bfdcd3919 owner: scheduled_skanta@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 4581 sha1: 3c96dfb3f50b3d02598a82af7104764bfdcd3919 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath20-testing-2026-02-13-0906-squid suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_3c96dfb3f50b3d02598a82af7104764bfdcd3919/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 3c96dfb3f50b3d02598a82af7104764bfdcd3919 targets: trial024.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBICjGajdMpoZZwL8Eo5R3NDRAZqDvTuH2zl+loisN7xHtg/sOg9BSJM6RNo+floVcuchuWC3YPR/W4lB0Eiwcps= tasks: - install: null - ceph: conf: osd: osd max pg log entries: 5 osd min pg log entries: 5 create_rbd_pool: false log-ignorelist: - \(OBJECT_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(SLOW_OPS\) - deep-scrub - missing - overall HEALTH_ - repair - slow request - unfound - \(POOL_APP_NOT_ENABLED\) - enough copies available pre-mgr-commands: - sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force - ec_inconsistent_hinfo: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-17_04:10:36 tube: trial user: skanta verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3682 2026-02-17T15:08:06.438 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_3c96dfb3f50b3d02598a82af7104764bfdcd3919/qa; will attempt to use it 2026-02-17T15:08:06.438 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_3c96dfb3f50b3d02598a82af7104764bfdcd3919/qa/tasks 2026-02-17T15:08:06.438 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-17T15:08:06.440 INFO:teuthology.task.internal:Checking packages... 2026-02-17T15:08:06.440 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '3c96dfb3f50b3d02598a82af7104764bfdcd3919' 2026-02-17T15:08:06.440 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-17T15:08:06.440 INFO:teuthology.packaging:ref: None 2026-02-17T15:08:06.440 INFO:teuthology.packaging:tag: None 2026-02-17T15:08:06.440 INFO:teuthology.packaging:branch: wip-bharath20-testing-2026-02-13-0906-squid 2026-02-17T15:08:06.440 INFO:teuthology.packaging:sha1: 3c96dfb3f50b3d02598a82af7104764bfdcd3919 2026-02-17T15:08:06.440 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-bharath20-testing-2026-02-13-0906-squid 2026-02-17T15:08:06.566 INFO:teuthology.task.internal:Found packages for ceph version 19.2.3-675-g3c96dfb3-1jammy 2026-02-17T15:08:06.567 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-17T15:08:06.580 INFO:teuthology.task.internal:no buildpackages task found 2026-02-17T15:08:06.580 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-17T15:08:06.593 INFO:teuthology.task.internal:Saving configuration 2026-02-17T15:08:06.604 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-17T15:08:06.617 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-17T15:08:06.694 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial024.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/skanta-2026-02-17_04:10:36-rados-wip-bharath20-testing-2026-02-13-0906-squid-distro-default-trial/51568', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-17 15:04:38.729913', 'locked_by': 'scheduled_skanta@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIJT+8A+rHU3Q5OMT29Jvn9oDaUY478mRnilVSk7w31B'} 2026-02-17T15:08:06.694 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-17T15:08:06.706 INFO:teuthology.task.internal:roles: ubuntu@trial024.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2026-02-17T15:08:06.706 INFO:teuthology.run_tasks:Running task console_log... 2026-02-17T15:08:06.791 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x706b70852340>, signals=[15]) 2026-02-17T15:08:06.792 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-17T15:08:06.803 INFO:teuthology.task.internal:Opening connections... 2026-02-17T15:08:06.803 DEBUG:teuthology.task.internal:connecting to ubuntu@trial024.front.sepia.ceph.com 2026-02-17T15:08:06.804 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial024.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-17T15:08:06.865 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-17T15:08:06.877 DEBUG:teuthology.orchestra.run.trial024:> uname -m 2026-02-17T15:08:06.941 INFO:teuthology.orchestra.run.trial024.stdout:x86_64 2026-02-17T15:08:06.942 DEBUG:teuthology.orchestra.run.trial024:> cat /etc/os-release 2026-02-17T15:08:06.989 INFO:teuthology.orchestra.run.trial024.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-17T15:08:06.989 INFO:teuthology.orchestra.run.trial024.stdout:NAME="Ubuntu" 2026-02-17T15:08:06.989 INFO:teuthology.orchestra.run.trial024.stdout:VERSION_ID="22.04" 2026-02-17T15:08:06.989 INFO:teuthology.orchestra.run.trial024.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-17T15:08:06.989 INFO:teuthology.orchestra.run.trial024.stdout:VERSION_CODENAME=jammy 2026-02-17T15:08:06.989 INFO:teuthology.orchestra.run.trial024.stdout:ID=ubuntu 2026-02-17T15:08:06.989 INFO:teuthology.orchestra.run.trial024.stdout:ID_LIKE=debian 2026-02-17T15:08:06.989 INFO:teuthology.orchestra.run.trial024.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-17T15:08:06.990 INFO:teuthology.orchestra.run.trial024.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-17T15:08:06.990 INFO:teuthology.orchestra.run.trial024.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-17T15:08:06.990 INFO:teuthology.orchestra.run.trial024.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-17T15:08:06.990 INFO:teuthology.orchestra.run.trial024.stdout:UBUNTU_CODENAME=jammy 2026-02-17T15:08:06.990 INFO:teuthology.lock.ops:Updating trial024.front.sepia.ceph.com on lock server 2026-02-17T15:08:07.081 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-17T15:08:07.094 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-17T15:08:07.106 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-17T15:08:07.106 DEBUG:teuthology.orchestra.run.trial024:> test '!' -e /home/ubuntu/cephtest 2026-02-17T15:08:07.110 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-17T15:08:07.121 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-17T15:08:07.121 DEBUG:teuthology.orchestra.run.trial024:> test -z $(ls -A /var/lib/ceph) 2026-02-17T15:08:07.162 INFO:teuthology.orchestra.run.trial024.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-17T15:08:07.163 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-17T15:08:07.263 INFO:teuthology.run_tasks:Running task kernel... 2026-02-17T15:08:07.283 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-17T15:08:07.283 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-17T15:08:07.283 DEBUG:teuthology.orchestra.run.trial024:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-17T15:08:07.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-17T15:08:07.287 DEBUG:teuthology.orchestra.run.trial024:> uname -r 2026-02-17T15:08:07.337 INFO:teuthology.orchestra.run.trial024.stdout:5.15.0-164-generic 2026-02-17T15:08:07.337 INFO:teuthology.task.kernel:Running kernel on trial024: 5.15.0-164-generic 2026-02-17T15:08:07.337 DEBUG:teuthology.orchestra.run.trial024:> sudo apt-get clean 2026-02-17T15:08:07.396 DEBUG:teuthology.orchestra.run.trial024:> sudo apt-get update 2026-02-17T15:08:07.591 INFO:teuthology.orchestra.run.trial024.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-17T15:08:07.599 INFO:teuthology.orchestra.run.trial024.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-17T15:08:07.637 INFO:teuthology.orchestra.run.trial024.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-17T15:08:07.653 INFO:teuthology.orchestra.run.trial024.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-17T15:08:07.735 INFO:teuthology.orchestra.run.trial024.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3236 kB] 2026-02-17T15:08:07.801 INFO:teuthology.orchestra.run.trial024.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [493 kB] 2026-02-17T15:08:07.804 INFO:teuthology.orchestra.run.trial024.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-17T15:08:07.805 INFO:teuthology.orchestra.run.trial024.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1255 kB] 2026-02-17T15:08:07.812 INFO:teuthology.orchestra.run.trial024.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [313 kB] 2026-02-17T15:08:07.814 INFO:teuthology.orchestra.run.trial024.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-17T15:08:07.814 INFO:teuthology.orchestra.run.trial024.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5287 kB] 2026-02-17T15:08:07.842 INFO:teuthology.orchestra.run.trial024.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [997 kB] 2026-02-17T15:08:07.845 INFO:teuthology.orchestra.run.trial024.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2966 kB] 2026-02-17T15:08:07.859 INFO:teuthology.orchestra.run.trial024.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [426 kB] 2026-02-17T15:08:07.860 INFO:teuthology.orchestra.run.trial024.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-17T15:08:07.860 INFO:teuthology.orchestra.run.trial024.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1017 kB] 2026-02-17T15:08:07.867 INFO:teuthology.orchestra.run.trial024.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe Translation-en [224 kB] 2026-02-17T15:08:07.867 INFO:teuthology.orchestra.run.trial024.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.6 kB] 2026-02-17T15:08:07.868 INFO:teuthology.orchestra.run.trial024.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5096 kB] 2026-02-17T15:08:07.886 INFO:teuthology.orchestra.run.trial024.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [964 kB] 2026-02-17T15:08:11.574 INFO:teuthology.orchestra.run.trial024.stdout:Fetched 22.7 MB in 2s (13.6 MB/s) 2026-02-17T15:08:12.105 INFO:teuthology.orchestra.run.trial024.stdout:Reading package lists... 2026-02-17T15:08:12.119 DEBUG:teuthology.orchestra.run.trial024:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-17T15:08:12.151 INFO:teuthology.orchestra.run.trial024.stdout:Reading package lists... 2026-02-17T15:08:12.260 INFO:teuthology.orchestra.run.trial024.stdout:Building dependency tree... 2026-02-17T15:08:12.261 INFO:teuthology.orchestra.run.trial024.stdout:Reading state information... 2026-02-17T15:08:12.347 INFO:teuthology.orchestra.run.trial024.stdout:The following packages were automatically installed and are no longer required: 2026-02-17T15:08:12.347 INFO:teuthology.orchestra.run.trial024.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-17T15:08:12.347 INFO:teuthology.orchestra.run.trial024.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-17T15:08:12.348 INFO:teuthology.orchestra.run.trial024.stdout: sg3-utils-udev 2026-02-17T15:08:12.348 INFO:teuthology.orchestra.run.trial024.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-17T15:08:12.348 INFO:teuthology.orchestra.run.trial024.stdout:The following additional packages will be installed: 2026-02-17T15:08:12.348 INFO:teuthology.orchestra.run.trial024.stdout: linux-generic linux-headers-5.15.0-170 linux-headers-5.15.0-170-generic 2026-02-17T15:08:12.348 INFO:teuthology.orchestra.run.trial024.stdout: linux-headers-generic linux-image-5.15.0-170-generic 2026-02-17T15:08:12.348 INFO:teuthology.orchestra.run.trial024.stdout: linux-modules-5.15.0-170-generic linux-modules-extra-5.15.0-170-generic 2026-02-17T15:08:12.348 INFO:teuthology.orchestra.run.trial024.stdout:Suggested packages: 2026-02-17T15:08:12.349 INFO:teuthology.orchestra.run.trial024.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2026-02-17T15:08:12.361 INFO:teuthology.orchestra.run.trial024.stdout:The following NEW packages will be installed: 2026-02-17T15:08:12.361 INFO:teuthology.orchestra.run.trial024.stdout: linux-headers-5.15.0-170 linux-headers-5.15.0-170-generic 2026-02-17T15:08:12.361 INFO:teuthology.orchestra.run.trial024.stdout: linux-image-5.15.0-170-generic linux-modules-5.15.0-170-generic 2026-02-17T15:08:12.361 INFO:teuthology.orchestra.run.trial024.stdout: linux-modules-extra-5.15.0-170-generic 2026-02-17T15:08:12.362 INFO:teuthology.orchestra.run.trial024.stdout:The following packages will be upgraded: 2026-02-17T15:08:12.362 INFO:teuthology.orchestra.run.trial024.stdout: linux-generic linux-headers-generic linux-image-generic 2026-02-17T15:08:12.456 INFO:teuthology.orchestra.run.trial024.stdout:3 upgraded, 5 newly installed, 0 to remove and 51 not upgraded. 2026-02-17T15:08:12.457 INFO:teuthology.orchestra.run.trial024.stdout:Need to get 113 MB of archives. 2026-02-17T15:08:12.457 INFO:teuthology.orchestra.run.trial024.stdout:After this operation, 585 MB of additional disk space will be used. 2026-02-17T15:08:12.457 INFO:teuthology.orchestra.run.trial024.stdout:Get:1 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-170-generic amd64 5.15.0-170.180 [22.7 MB] 2026-02-17T15:08:12.626 INFO:teuthology.orchestra.run.trial024.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-170-generic amd64 5.15.0-170.180 [11.6 MB] 2026-02-17T15:08:12.670 INFO:teuthology.orchestra.run.trial024.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-170-generic amd64 5.15.0-170.180 [64.0 MB] 2026-02-17T15:08:12.906 INFO:teuthology.orchestra.run.trial024.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.170.159 [1654 B] 2026-02-17T15:08:12.906 INFO:teuthology.orchestra.run.trial024.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.170.159 [2474 B] 2026-02-17T15:08:12.906 INFO:teuthology.orchestra.run.trial024.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-170 all 5.15.0-170.180 [12.4 MB] 2026-02-17T15:08:12.954 INFO:teuthology.orchestra.run.trial024.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-170-generic amd64 5.15.0-170.180 [2828 kB] 2026-02-17T15:08:12.964 INFO:teuthology.orchestra.run.trial024.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.170.159 [2316 B] 2026-02-17T15:08:13.163 INFO:teuthology.orchestra.run.trial024.stdout:Fetched 113 MB in 1s (189 MB/s) 2026-02-17T15:08:13.180 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package linux-modules-5.15.0-170-generic. 2026-02-17T15:08:13.345 INFO:teuthology.orchestra.run.trial024.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 131954 files and directories currently installed.) 2026-02-17T15:08:13.347 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../0-linux-modules-5.15.0-170-generic_5.15.0-170.180_amd64.deb ... 2026-02-17T15:08:13.348 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking linux-modules-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-17T15:08:13.702 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package linux-image-5.15.0-170-generic. 2026-02-17T15:08:13.708 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../1-linux-image-5.15.0-170-generic_5.15.0-170.180_amd64.deb ... 2026-02-17T15:08:13.726 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking linux-image-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-17T15:08:13.767 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-170-generic. 2026-02-17T15:08:13.773 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-170-generic_5.15.0-170.180_amd64.deb ... 2026-02-17T15:08:13.774 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking linux-modules-extra-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-17T15:08:14.884 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../3-linux-generic_5.15.0.170.159_amd64.deb ... 2026-02-17T15:08:14.885 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking linux-generic (5.15.0.170.159) over (5.15.0.164.159) ... 2026-02-17T15:08:14.906 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.170.159_amd64.deb ... 2026-02-17T15:08:14.907 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking linux-image-generic (5.15.0.170.159) over (5.15.0.164.159) ... 2026-02-17T15:08:14.923 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package linux-headers-5.15.0-170. 2026-02-17T15:08:14.930 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../5-linux-headers-5.15.0-170_5.15.0-170.180_all.deb ... 2026-02-17T15:08:14.930 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking linux-headers-5.15.0-170 (5.15.0-170.180) ... 2026-02-17T15:08:16.310 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package linux-headers-5.15.0-170-generic. 2026-02-17T15:08:16.321 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../6-linux-headers-5.15.0-170-generic_5.15.0-170.180_amd64.deb ... 2026-02-17T15:08:16.322 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking linux-headers-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-17T15:08:16.913 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.170.159_amd64.deb ... 2026-02-17T15:08:16.914 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking linux-headers-generic (5.15.0.170.159) over (5.15.0.164.159) ... 2026-02-17T15:08:16.939 INFO:teuthology.orchestra.run.trial024.stdout:Setting up linux-headers-5.15.0-170 (5.15.0-170.180) ... 2026-02-17T15:08:16.940 INFO:teuthology.orchestra.run.trial024.stdout:Setting up linux-headers-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-17T15:08:16.941 INFO:teuthology.orchestra.run.trial024.stdout:Setting up linux-headers-generic (5.15.0.170.159) ... 2026-02-17T15:08:16.943 INFO:teuthology.orchestra.run.trial024.stdout:Setting up linux-modules-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-17T15:08:17.712 INFO:teuthology.orchestra.run.trial024.stdout:Setting up linux-image-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-17T15:08:18.534 INFO:teuthology.orchestra.run.trial024.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.15.0-170-generic 2026-02-17T15:08:18.534 INFO:teuthology.orchestra.run.trial024.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.15.0-170-generic 2026-02-17T15:08:18.541 INFO:teuthology.orchestra.run.trial024.stdout:Setting up linux-modules-extra-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-17T15:08:19.331 INFO:teuthology.orchestra.run.trial024.stdout:Setting up linux-image-generic (5.15.0.170.159) ... 2026-02-17T15:08:19.332 INFO:teuthology.orchestra.run.trial024.stdout:Setting up linux-generic (5.15.0.170.159) ... 2026-02-17T15:08:19.334 INFO:teuthology.orchestra.run.trial024.stdout:Processing triggers for linux-image-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-17T15:08:19.339 INFO:teuthology.orchestra.run.trial024.stdout:/etc/kernel/postinst.d/initramfs-tools: 2026-02-17T15:08:19.339 INFO:teuthology.orchestra.run.trial024.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-170-generic 2026-02-17T15:08:26.438 INFO:teuthology.orchestra.run.trial024.stdout:W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast 2026-02-17T15:08:33.061 INFO:teuthology.orchestra.run.trial024.stdout:/etc/kernel/postinst.d/zz-update-grub: 2026-02-17T15:08:33.061 INFO:teuthology.orchestra.run.trial024.stdout:Sourcing file `/etc/default/grub' 2026-02-17T15:08:33.080 INFO:teuthology.orchestra.run.trial024.stdout:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-17T15:08:33.081 INFO:teuthology.orchestra.run.trial024.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-17T15:08:33.083 INFO:teuthology.orchestra.run.trial024.stdout:Generating grub configuration file ... 2026-02-17T15:08:33.166 INFO:teuthology.orchestra.run.trial024.stdout:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-17T15:08:33.172 INFO:teuthology.orchestra.run.trial024.stdout:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-17T15:08:33.345 INFO:teuthology.orchestra.run.trial024.stdout:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-17T15:08:33.350 INFO:teuthology.orchestra.run.trial024.stdout:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-17T15:08:33.423 INFO:teuthology.orchestra.run.trial024.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-17T15:08:33.423 INFO:teuthology.orchestra.run.trial024.stdout:Systems on them will not be added to the GRUB boot configuration. 2026-02-17T15:08:33.423 INFO:teuthology.orchestra.run.trial024.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-17T15:08:33.426 INFO:teuthology.orchestra.run.trial024.stdout:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-17T15:08:33.434 INFO:teuthology.orchestra.run.trial024.stdout:done 2026-02-17T15:08:33.822 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:08:33.823 INFO:teuthology.orchestra.run.trial024.stdout:Pending kernel upgrade! 2026-02-17T15:08:33.823 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:08:33.823 INFO:teuthology.orchestra.run.trial024.stdout:Running kernel version: 2026-02-17T15:08:33.823 INFO:teuthology.orchestra.run.trial024.stdout: 5.15.0-164-generic 2026-02-17T15:08:33.823 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:08:33.823 INFO:teuthology.orchestra.run.trial024.stdout:Diagnostics: 2026-02-17T15:08:33.823 INFO:teuthology.orchestra.run.trial024.stdout: The currently running kernel version is not the expected kernel version 5.15.0-170-generic. 2026-02-17T15:08:33.823 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:08:33.823 INFO:teuthology.orchestra.run.trial024.stdout:Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting. [Return] 2026-02-17T15:08:33.823 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:08:33.824 INFO:teuthology.orchestra.run.trial024.stdout:The processor microcode seems to be up-to-date. 2026-02-17T15:08:33.824 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:08:33.824 INFO:teuthology.orchestra.run.trial024.stdout:No services need to be restarted. 2026-02-17T15:08:33.824 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:08:33.824 INFO:teuthology.orchestra.run.trial024.stdout:No containers need to be restarted. 2026-02-17T15:08:33.824 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:08:33.824 INFO:teuthology.orchestra.run.trial024.stdout:No user sessions are running outdated binaries. 2026-02-17T15:08:33.824 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:08:33.824 INFO:teuthology.orchestra.run.trial024.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-17T15:08:34.466 DEBUG:teuthology.orchestra.run.trial024:> dpkg -s linux-image-generic 2026-02-17T15:08:34.477 INFO:teuthology.orchestra.run.trial024.stdout:Package: linux-image-generic 2026-02-17T15:08:34.477 INFO:teuthology.orchestra.run.trial024.stdout:Status: install ok installed 2026-02-17T15:08:34.477 INFO:teuthology.orchestra.run.trial024.stdout:Priority: optional 2026-02-17T15:08:34.477 INFO:teuthology.orchestra.run.trial024.stdout:Section: kernel 2026-02-17T15:08:34.477 INFO:teuthology.orchestra.run.trial024.stdout:Installed-Size: 21 2026-02-17T15:08:34.477 INFO:teuthology.orchestra.run.trial024.stdout:Maintainer: Ubuntu Kernel Team 2026-02-17T15:08:34.478 INFO:teuthology.orchestra.run.trial024.stdout:Architecture: amd64 2026-02-17T15:08:34.478 INFO:teuthology.orchestra.run.trial024.stdout:Source: linux-meta 2026-02-17T15:08:34.478 INFO:teuthology.orchestra.run.trial024.stdout:Version: 5.15.0.170.159 2026-02-17T15:08:34.478 INFO:teuthology.orchestra.run.trial024.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-17T15:08:34.478 INFO:teuthology.orchestra.run.trial024.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-17T15:08:34.478 INFO:teuthology.orchestra.run.trial024.stdout:Recommends: thermald 2026-02-17T15:08:34.478 INFO:teuthology.orchestra.run.trial024.stdout:Description: Generic Linux kernel image 2026-02-17T15:08:34.478 INFO:teuthology.orchestra.run.trial024.stdout: This package will always depend on the latest generic kernel image 2026-02-17T15:08:34.478 INFO:teuthology.orchestra.run.trial024.stdout: available. 2026-02-17T15:08:34.479 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-164-generic Expected: 5.15.0-170-generic 2026-02-17T15:08:34.479 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-17T15:08:34.479 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-17T15:08:34.479 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-17T15:08:34.479 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial024.front.sepia.ceph.com, path=None, version=distro) 2026-02-17T15:08:34.479 DEBUG:teuthology.orchestra.run.trial024:> sudo apt-get clean 2026-02-17T15:08:34.560 DEBUG:teuthology.orchestra.run.trial024:> sudo apt-get update 2026-02-17T15:08:34.832 INFO:teuthology.orchestra.run.trial024.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-17T15:08:34.836 INFO:teuthology.orchestra.run.trial024.stdout:Hit:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-17T15:08:34.846 INFO:teuthology.orchestra.run.trial024.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-17T15:08:34.856 INFO:teuthology.orchestra.run.trial024.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-17T15:08:35.700 INFO:teuthology.orchestra.run.trial024.stdout:Reading package lists... 2026-02-17T15:08:35.715 DEBUG:teuthology.orchestra.run.trial024:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-17T15:08:35.790 INFO:teuthology.orchestra.run.trial024.stdout:Reading package lists... 2026-02-17T15:08:35.901 INFO:teuthology.orchestra.run.trial024.stdout:Building dependency tree... 2026-02-17T15:08:35.901 INFO:teuthology.orchestra.run.trial024.stdout:Reading state information... 2026-02-17T15:08:35.986 INFO:teuthology.orchestra.run.trial024.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-17T15:08:35.986 INFO:teuthology.orchestra.run.trial024.stdout:linux-image-generic set to manually installed. 2026-02-17T15:08:35.986 INFO:teuthology.orchestra.run.trial024.stdout:The following packages were automatically installed and are no longer required: 2026-02-17T15:08:35.986 INFO:teuthology.orchestra.run.trial024.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-17T15:08:35.986 INFO:teuthology.orchestra.run.trial024.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-17T15:08:35.987 INFO:teuthology.orchestra.run.trial024.stdout: sg3-utils-udev 2026-02-17T15:08:35.987 INFO:teuthology.orchestra.run.trial024.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-17T15:08:36.006 INFO:teuthology.orchestra.run.trial024.stdout:0 upgraded, 0 newly installed, 0 to remove and 51 not upgraded. 2026-02-17T15:08:36.008 DEBUG:teuthology.orchestra.run.trial024:> dpkg -s linux-image-generic 2026-02-17T15:08:36.058 INFO:teuthology.orchestra.run.trial024.stdout:Package: linux-image-generic 2026-02-17T15:08:36.059 INFO:teuthology.orchestra.run.trial024.stdout:Status: install ok installed 2026-02-17T15:08:36.059 INFO:teuthology.orchestra.run.trial024.stdout:Priority: optional 2026-02-17T15:08:36.059 INFO:teuthology.orchestra.run.trial024.stdout:Section: kernel 2026-02-17T15:08:36.059 INFO:teuthology.orchestra.run.trial024.stdout:Installed-Size: 21 2026-02-17T15:08:36.059 INFO:teuthology.orchestra.run.trial024.stdout:Maintainer: Ubuntu Kernel Team 2026-02-17T15:08:36.059 INFO:teuthology.orchestra.run.trial024.stdout:Architecture: amd64 2026-02-17T15:08:36.059 INFO:teuthology.orchestra.run.trial024.stdout:Source: linux-meta 2026-02-17T15:08:36.059 INFO:teuthology.orchestra.run.trial024.stdout:Version: 5.15.0.170.159 2026-02-17T15:08:36.059 INFO:teuthology.orchestra.run.trial024.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-17T15:08:36.059 INFO:teuthology.orchestra.run.trial024.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-17T15:08:36.060 INFO:teuthology.orchestra.run.trial024.stdout:Recommends: thermald 2026-02-17T15:08:36.060 INFO:teuthology.orchestra.run.trial024.stdout:Description: Generic Linux kernel image 2026-02-17T15:08:36.060 INFO:teuthology.orchestra.run.trial024.stdout: This package will always depend on the latest generic kernel image 2026-02-17T15:08:36.060 INFO:teuthology.orchestra.run.trial024.stdout: available. 2026-02-17T15:08:36.061 DEBUG:teuthology.orchestra.run.trial024:> mktemp 2026-02-17T15:08:36.106 INFO:teuthology.orchestra.run.trial024.stdout:/tmp/tmp.oglRktFm4f 2026-02-17T15:08:36.106 DEBUG:teuthology.orchestra.run.trial024:> sudo cp /boot/grub/grub.cfg /tmp/tmp.oglRktFm4f 2026-02-17T15:08:36.154 DEBUG:teuthology.orchestra.run.trial024:> sudo chmod 0666 /tmp/tmp.oglRktFm4f 2026-02-17T15:08:36.256 DEBUG:teuthology.orchestra.remote:trial024:/tmp/tmp.oglRktFm4f is 7KB 2026-02-17T15:08:36.264 DEBUG:teuthology.orchestra.run.trial024:> rm -fr /tmp/tmp.oglRktFm4f 2026-02-17T15:08:36.268 DEBUG:teuthology.orchestra.run.trial024:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2026-02-17T15:08:36.318 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:08:36.318 DEBUG:teuthology.orchestra.run.trial024:> sudo dd of=/etc/grub.d/01_ceph_kernel 2026-02-17T15:08:36.318 DEBUG:teuthology.orchestra.run.trial024:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2026-02-17T15:08:36.372 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-170-generic 2026-02-17T15:08:36.372 DEBUG:teuthology.orchestra.run.trial024:> sudo update-grub 2026-02-17T15:08:36.614 INFO:teuthology.orchestra.run.trial024.stderr:Sourcing file `/etc/default/grub' 2026-02-17T15:08:36.634 INFO:teuthology.orchestra.run.trial024.stderr:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-17T15:08:36.635 INFO:teuthology.orchestra.run.trial024.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-17T15:08:36.637 INFO:teuthology.orchestra.run.trial024.stderr:Generating grub configuration file ... 2026-02-17T15:08:36.720 INFO:teuthology.orchestra.run.trial024.stderr:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-17T15:08:36.726 INFO:teuthology.orchestra.run.trial024.stderr:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-17T15:08:36.903 INFO:teuthology.orchestra.run.trial024.stderr:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-17T15:08:36.909 INFO:teuthology.orchestra.run.trial024.stderr:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-17T15:08:36.977 INFO:teuthology.orchestra.run.trial024.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-17T15:08:36.977 INFO:teuthology.orchestra.run.trial024.stderr:Systems on them will not be added to the GRUB boot configuration. 2026-02-17T15:08:36.977 INFO:teuthology.orchestra.run.trial024.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-17T15:08:36.979 INFO:teuthology.orchestra.run.trial024.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-17T15:08:36.986 INFO:teuthology.orchestra.run.trial024.stderr:done 2026-02-17T15:08:36.988 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial024 2026-02-17T15:08:36.988 DEBUG:teuthology.orchestra.run.trial024:> sudo bash -c ' 2026-02-17T15:08:36.988 DEBUG:teuthology.orchestra.run.trial024:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-17T15:08:36.988 DEBUG:teuthology.orchestra.run.trial024:> ' 2026-02-17T15:09:06.991 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-17T15:09:06.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial024.front.sepia.ceph.com' 2026-02-17T15:09:06.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial024.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-17T15:10:07.051 DEBUG:teuthology.orchestra.remote:timed out 2026-02-17T15:10:16.058 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial024.front.sepia.ceph.com' 2026-02-17T15:10:16.059 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial024.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-17T15:10:16.186 DEBUG:teuthology.orchestra.run.trial024:> true 2026-02-17T15:10:16.584 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial024.front.sepia.ceph.com' 2026-02-17T15:10:16.584 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-170-generic"... 2026-02-17T15:10:16.585 DEBUG:teuthology.orchestra.run.trial024:> uname -r 2026-02-17T15:10:16.630 INFO:teuthology.orchestra.run.trial024.stdout:5.15.0-170-generic 2026-02-17T15:10:16.630 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-170-generic vs 5.15.0-170-generic 2026-02-17T15:10:16.630 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-17T15:10:16.630 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2026-02-17T15:10:17.631 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-17T15:10:17.631 DEBUG:teuthology.orchestra.run.trial024:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-17T15:10:17.644 INFO:teuthology.orchestra.run.trial024.stdout:ttyS1 2026-02-17T15:10:17.668 DEBUG:teuthology.parallel:result is None 2026-02-17T15:10:17.668 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-17T15:10:17.680 INFO:teuthology.task.internal:Creating test directory... 2026-02-17T15:10:17.681 DEBUG:teuthology.orchestra.run.trial024:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-17T15:10:17.714 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-17T15:10:17.729 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-17T15:10:17.742 INFO:teuthology.task.internal:Creating archive directory... 2026-02-17T15:10:17.743 DEBUG:teuthology.orchestra.run.trial024:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-17T15:10:17.763 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-17T15:10:17.775 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-17T15:10:17.775 DEBUG:teuthology.orchestra.run.trial024:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-17T15:10:17.810 INFO:teuthology.orchestra.run.trial024.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-17T15:10:17.817 INFO:teuthology.orchestra.run.trial024.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-17T15:10:17.818 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-17T15:10:17.831 INFO:teuthology.task.internal:Configuring sudo... 2026-02-17T15:10:17.831 DEBUG:teuthology.orchestra.run.trial024:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-17T15:10:17.870 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-17T15:10:17.886 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-17T15:10:17.886 DEBUG:teuthology.orchestra.run.trial024:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-17T15:10:17.918 DEBUG:teuthology.orchestra.run.trial024:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-17T15:10:17.962 DEBUG:teuthology.orchestra.run.trial024:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-17T15:10:18.006 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:10:18.006 DEBUG:teuthology.orchestra.run.trial024:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-17T15:10:18.059 DEBUG:teuthology.orchestra.run.trial024:> sudo service rsyslog restart 2026-02-17T15:10:18.119 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-17T15:10:18.132 INFO:teuthology.task.internal:Starting timer... 2026-02-17T15:10:18.132 INFO:teuthology.run_tasks:Running task pcp... 2026-02-17T15:10:18.154 INFO:teuthology.run_tasks:Running task selinux... 2026-02-17T15:10:18.171 DEBUG:teuthology.task.selinux:Excluding trial024: OS 'ubuntu' does not support SELinux 2026-02-17T15:10:18.172 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-17T15:10:18.172 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-17T15:10:18.172 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-17T15:10:18.172 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-17T15:10:18.187 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-17T15:10:18.201 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-17T15:10:18.202 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial024.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-17T15:12:45.372 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial024.front.sepia.ceph.com')] 2026-02-17T15:12:45.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial024.front.sepia.ceph.com' 2026-02-17T15:12:45.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial024.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-17T15:12:45.485 DEBUG:teuthology.orchestra.run.trial024:> true 2026-02-17T15:12:45.737 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial024.front.sepia.ceph.com' 2026-02-17T15:12:45.737 INFO:teuthology.run_tasks:Running task clock... 2026-02-17T15:12:45.757 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-17T15:12:45.757 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-17T15:12:45.758 DEBUG:teuthology.orchestra.run.trial024:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-17T15:12:45.800 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-17T15:12:45.800 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: Command line: ntpd -gq 2026-02-17T15:12:45.800 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: ---------------------------------------------------- 2026-02-17T15:12:45.801 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: ntp-4 is maintained by Network Time Foundation, 2026-02-17T15:12:45.801 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-17T15:12:45.801 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: corporation. Support and training for ntp-4 are 2026-02-17T15:12:45.801 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: available at https://www.nwtime.org/support 2026-02-17T15:12:45.801 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: ---------------------------------------------------- 2026-02-17T15:12:45.801 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: proto: precision = 0.030 usec (-25) 2026-02-17T15:12:45.801 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: basedate set to 2022-02-04 2026-02-17T15:12:45.801 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: gps base set to 2022-02-06 (week 2196) 2026-02-17T15:12:45.801 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-17T15:12:45.802 INFO:teuthology.orchestra.run.trial024.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-17T15:12:45.802 INFO:teuthology.orchestra.run.trial024.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-17T15:12:45.803 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: restrict ::: KOD does nothing without LIMITED. 2026-02-17T15:12:45.803 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: Listen and drop on 0 v6wildcard [::]:123 2026-02-17T15:12:45.803 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-17T15:12:45.803 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: Listen normally on 2 lo 127.0.0.1:123 2026-02-17T15:12:45.803 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: Listen normally on 3 enp1s0f0 10.20.193.24:123 2026-02-17T15:12:45.803 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: Listen normally on 4 lo [::1]:123 2026-02-17T15:12:45.803 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:6100%2]:123 2026-02-17T15:12:45.803 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:42 ntpd[8789]: Listening on routing socket on fd #22 for interface updates 2026-02-17T15:12:46.800 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:43 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:46.800 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:43 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:46.801 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:43 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:46.801 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:43 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:46.813 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:43 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:46.813 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:43 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:46.813 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:43 ntpd[8789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-17T15:12:46.813 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:43 ntpd[8789]: can't open /var/log/ntpstats/peerstats.20260217: Permission denied 2026-02-17T15:12:46.910 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:43 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:46.910 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:43 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:46.910 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:43 ntpd[8789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-17T15:12:46.910 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:43 ntpd[8789]: can't open /var/log/ntpstats/peerstats.20260217: Permission denied 2026-02-17T15:12:48.800 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:45 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:48.800 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:45 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:48.800 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:45 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:48.801 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:45 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:48.813 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:45 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:48.813 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:45 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:48.813 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:45 ntpd[8789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-17T15:12:48.814 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:45 ntpd[8789]: can't open /var/log/ntpstats/peerstats.20260217: Permission denied 2026-02-17T15:12:48.909 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:45 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:48.909 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:45 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:48.909 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:45 ntpd[8789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-17T15:12:48.909 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:45 ntpd[8789]: can't open /var/log/ntpstats/peerstats.20260217: Permission denied 2026-02-17T15:12:49.800 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:46 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:49.800 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:46 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:49.869 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:46 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:49.869 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:46 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:49.870 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:46 ntpd[8789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-17T15:12:49.870 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:46 ntpd[8789]: can't open /var/log/ntpstats/peerstats.20260217: Permission denied 2026-02-17T15:12:50.800 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:50.801 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:50.801 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:50.801 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:50.801 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:50.801 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:50.803 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:50.803 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:50.804 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-17T15:12:50.804 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: can't open /var/log/ntpstats/peerstats.20260217: Permission denied 2026-02-17T15:12:50.813 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:50.813 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:50.813 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-17T15:12:50.814 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: can't open /var/log/ntpstats/peerstats.20260217: Permission denied 2026-02-17T15:12:50.909 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:50.910 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:50.910 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-17T15:12:50.910 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:47 ntpd[8789]: can't open /var/log/ntpstats/peerstats.20260217: Permission denied 2026-02-17T15:12:51.800 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:48 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:51.800 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:48 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:51.869 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:48 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:51.869 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:48 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:51.869 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:48 ntpd[8789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-17T15:12:51.869 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:48 ntpd[8789]: can't open /var/log/ntpstats/peerstats.20260217: Permission denied 2026-02-17T15:12:52.800 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:49 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:52.801 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:49 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:52.801 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:49 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:52.801 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:49 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:52.801 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:49 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:52.801 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:49 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:52.803 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:49 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:52.803 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:49 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:52.803 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:49 ntpd[8789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-17T15:12:52.804 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:49 ntpd[8789]: can't open /var/log/ntpstats/peerstats.20260217: Permission denied 2026-02-17T15:12:52.813 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:49 ntpd[8789]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-17T15:12:52.813 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:49 ntpd[8789]: can't open /var/log/ntpstats/rawstats.20260217: Permission denied 2026-02-17T15:12:52.813 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:49 ntpd[8789]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-17T15:12:52.814 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:49 ntpd[8789]: can't open /var/log/ntpstats/peerstats.20260217: Permission denied 2026-02-17T15:12:52.814 INFO:teuthology.orchestra.run.trial024.stdout:17 Feb 15:12:52 ntpd[8789]: ntpd: time set +3.007518 s 2026-02-17T15:12:52.814 INFO:teuthology.orchestra.run.trial024.stdout:ntpd: time set +3.007518s 2026-02-17T15:12:52.814 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:52 ntpd[8789]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-17T15:12:52.815 INFO:teuthology.orchestra.run.trial024.stderr:17 Feb 15:12:52 ntpd[8789]: can't open /var/log/ntpstats/loopstats.20260217: Permission denied 2026-02-17T15:12:52.889 INFO:teuthology.orchestra.run.trial024.stdout: remote refid st t when poll reach delay offset jitter 2026-02-17T15:12:52.889 INFO:teuthology.orchestra.run.trial024.stdout:============================================================================== 2026-02-17T15:12:52.889 INFO:teuthology.orchestra.run.trial024.stdout: ntp2.zeus.frume .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-17T15:12:52.889 INFO:teuthology.orchestra.run.trial024.stdout: LAX.CALTICK.NET .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-17T15:12:52.889 INFO:teuthology.orchestra.run.trial024.stdout: 73.185.182.209 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-17T15:12:52.890 INFO:teuthology.orchestra.run.trial024.stdout: ntp.maxhost.io .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-17T15:12:52.891 INFO:teuthology.run_tasks:Running task install... 2026-02-17T15:12:52.906 DEBUG:teuthology.task.install:project ceph 2026-02-17T15:12:52.906 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3c96dfb3f50b3d02598a82af7104764bfdcd3919'}} 2026-02-17T15:12:52.906 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '3c96dfb3f50b3d02598a82af7104764bfdcd3919'} 2026-02-17T15:12:52.906 INFO:teuthology.task.install:Using flavor: default 2026-02-17T15:12:52.915 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-17T15:12:52.915 INFO:teuthology.task.install:extra packages: [] 2026-02-17T15:12:52.916 DEBUG:teuthology.orchestra.run.trial024:> sudo apt-key list | grep Ceph 2026-02-17T15:12:52.979 INFO:teuthology.orchestra.run.trial024.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-17T15:12:53.006 INFO:teuthology.orchestra.run.trial024.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-17T15:12:53.006 INFO:teuthology.orchestra.run.trial024.stdout:uid [ unknown] Ceph.com (release key) 2026-02-17T15:12:53.007 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-17T15:12:53.008 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3c96dfb3f50b3d02598a82af7104764bfdcd3919 2026-02-17T15:12:53.212 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default/ 2026-02-17T15:12:53.212 INFO:teuthology.task.install.deb:Package version is 19.2.3-675-g3c96dfb3-1jammy 2026-02-17T15:12:53.369 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:12:53.369 DEBUG:teuthology.orchestra.run.trial024:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-17T15:12:53.378 DEBUG:teuthology.orchestra.run.trial024:> sudo apt-get update 2026-02-17T15:12:53.547 INFO:teuthology.orchestra.run.trial024.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy InRelease 2026-02-17T15:12:53.575 INFO:teuthology.orchestra.run.trial024.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy Release [6978 B] 2026-02-17T15:12:53.586 INFO:teuthology.orchestra.run.trial024.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-17T15:12:53.640 INFO:teuthology.orchestra.run.trial024.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2026-02-17T15:12:53.818 INFO:teuthology.orchestra.run.trial024.stdout:Hit:5 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-17T15:12:53.885 INFO:teuthology.orchestra.run.trial024.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-17T15:12:53.957 INFO:teuthology.orchestra.run.trial024.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-17T15:12:54.029 INFO:teuthology.orchestra.run.trial024.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-17T15:12:54.174 INFO:teuthology.orchestra.run.trial024.stdout:Fetched 25.1 kB in 1s (39.4 kB/s) 2026-02-17T15:12:54.700 INFO:teuthology.orchestra.run.trial024.stdout:Reading package lists... 2026-02-17T15:12:54.714 DEBUG:teuthology.orchestra.run.trial024:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.2.3-675-g3c96dfb3-1jammy cephadm=19.2.3-675-g3c96dfb3-1jammy ceph-mds=19.2.3-675-g3c96dfb3-1jammy ceph-mgr=19.2.3-675-g3c96dfb3-1jammy ceph-common=19.2.3-675-g3c96dfb3-1jammy ceph-fuse=19.2.3-675-g3c96dfb3-1jammy ceph-test=19.2.3-675-g3c96dfb3-1jammy ceph-volume=19.2.3-675-g3c96dfb3-1jammy radosgw=19.2.3-675-g3c96dfb3-1jammy python3-rados=19.2.3-675-g3c96dfb3-1jammy python3-rgw=19.2.3-675-g3c96dfb3-1jammy python3-cephfs=19.2.3-675-g3c96dfb3-1jammy python3-rbd=19.2.3-675-g3c96dfb3-1jammy libcephfs2=19.2.3-675-g3c96dfb3-1jammy libcephfs-dev=19.2.3-675-g3c96dfb3-1jammy librados2=19.2.3-675-g3c96dfb3-1jammy librbd1=19.2.3-675-g3c96dfb3-1jammy rbd-fuse=19.2.3-675-g3c96dfb3-1jammy 2026-02-17T15:12:54.748 INFO:teuthology.orchestra.run.trial024.stdout:Reading package lists... 2026-02-17T15:12:54.859 INFO:teuthology.orchestra.run.trial024.stdout:Building dependency tree... 2026-02-17T15:12:54.860 INFO:teuthology.orchestra.run.trial024.stdout:Reading state information... 2026-02-17T15:12:54.951 INFO:teuthology.orchestra.run.trial024.stdout:The following packages were automatically installed and are no longer required: 2026-02-17T15:12:54.952 INFO:teuthology.orchestra.run.trial024.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-17T15:12:54.952 INFO:teuthology.orchestra.run.trial024.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-17T15:12:54.952 INFO:teuthology.orchestra.run.trial024.stdout: sg3-utils-udev 2026-02-17T15:12:54.952 INFO:teuthology.orchestra.run.trial024.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-17T15:12:54.952 INFO:teuthology.orchestra.run.trial024.stdout:The following additional packages will be installed: 2026-02-17T15:12:54.952 INFO:teuthology.orchestra.run.trial024.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-17T15:12:54.952 INFO:teuthology.orchestra.run.trial024.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-17T15:12:54.952 INFO:teuthology.orchestra.run.trial024.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 2026-02-17T15:12:54.953 INFO:teuthology.orchestra.run.trial024.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-17T15:12:54.953 INFO:teuthology.orchestra.run.trial024.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2026-02-17T15:12:54.953 INFO:teuthology.orchestra.run.trial024.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2026-02-17T15:12:54.953 INFO:teuthology.orchestra.run.trial024.stdout: pkg-config python-asyncssh-doc python-pastedeploy-tpl python3-asyncssh 2026-02-17T15:12:54.953 INFO:teuthology.orchestra.run.trial024.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cheroot 2026-02-17T15:12:54.953 INFO:teuthology.orchestra.run.trial024.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-17T15:12:54.953 INFO:teuthology.orchestra.run.trial024.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-17T15:12:54.953 INFO:teuthology.orchestra.run.trial024.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-17T15:12:54.953 INFO:teuthology.orchestra.run.trial024.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-17T15:12:54.953 INFO:teuthology.orchestra.run.trial024.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-17T15:12:54.953 INFO:teuthology.orchestra.run.trial024.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-17T15:12:54.953 INFO:teuthology.orchestra.run.trial024.stdout: python3-pyinotify python3-pytest python3-repoze.lru 2026-02-17T15:12:54.953 INFO:teuthology.orchestra.run.trial024.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2026-02-17T15:12:54.954 INFO:teuthology.orchestra.run.trial024.stdout: python3-simplejson python3-singledispatch python3-sklearn 2026-02-17T15:12:54.954 INFO:teuthology.orchestra.run.trial024.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2026-02-17T15:12:54.954 INFO:teuthology.orchestra.run.trial024.stdout: python3-toml python3-waitress python3-wcwidth python3-webob 2026-02-17T15:12:54.954 INFO:teuthology.orchestra.run.trial024.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2026-02-17T15:12:54.954 INFO:teuthology.orchestra.run.trial024.stdout: qttranslations5-l10n socat unzip xmlstarlet zip 2026-02-17T15:12:54.954 INFO:teuthology.orchestra.run.trial024.stdout:Suggested packages: 2026-02-17T15:12:54.954 INFO:teuthology.orchestra.run.trial024.stdout: python3-influxdb readline-doc python3-beaker python-mako-doc 2026-02-17T15:12:54.954 INFO:teuthology.orchestra.run.trial024.stdout: python-natsort-doc httpd-wsgi libapache2-mod-python libapache2-mod-scgi 2026-02-17T15:12:54.954 INFO:teuthology.orchestra.run.trial024.stdout: libjs-mochikit python-pecan-doc python-psutil-doc subversion 2026-02-17T15:12:54.954 INFO:teuthology.orchestra.run.trial024.stdout: python-pygments-doc ttf-bitstream-vera python-pyinotify-doc python3-dap 2026-02-17T15:12:54.954 INFO:teuthology.orchestra.run.trial024.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2026-02-17T15:12:54.954 INFO:teuthology.orchestra.run.trial024.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2026-02-17T15:12:54.954 INFO:teuthology.orchestra.run.trial024.stdout:Recommended packages: 2026-02-17T15:12:54.955 INFO:teuthology.orchestra.run.trial024.stdout: btrfs-tools 2026-02-17T15:12:54.985 INFO:teuthology.orchestra.run.trial024.stdout:The following NEW packages will be installed: 2026-02-17T15:12:54.985 INFO:teuthology.orchestra.run.trial024.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-17T15:12:54.986 INFO:teuthology.orchestra.run.trial024.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-17T15:12:54.986 INFO:teuthology.orchestra.run.trial024.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-17T15:12:54.986 INFO:teuthology.orchestra.run.trial024.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 2026-02-17T15:12:54.986 INFO:teuthology.orchestra.run.trial024.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-17T15:12:54.986 INFO:teuthology.orchestra.run.trial024.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2026-02-17T15:12:54.986 INFO:teuthology.orchestra.run.trial024.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2026-02-17T15:12:54.986 INFO:teuthology.orchestra.run.trial024.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2026-02-17T15:12:54.986 INFO:teuthology.orchestra.run.trial024.stdout: python-pastedeploy-tpl python3-asyncssh python3-cachetools 2026-02-17T15:12:54.986 INFO:teuthology.orchestra.run.trial024.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cheroot 2026-02-17T15:12:54.986 INFO:teuthology.orchestra.run.trial024.stdout: python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-17T15:12:54.986 INFO:teuthology.orchestra.run.trial024.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-17T15:12:54.987 INFO:teuthology.orchestra.run.trial024.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-logutils 2026-02-17T15:12:54.987 INFO:teuthology.orchestra.run.trial024.stdout: python3-mako python3-natsort python3-paste python3-pastedeploy 2026-02-17T15:12:54.987 INFO:teuthology.orchestra.run.trial024.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2026-02-17T15:12:54.987 INFO:teuthology.orchestra.run.trial024.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-17T15:12:54.987 INFO:teuthology.orchestra.run.trial024.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2026-02-17T15:12:54.987 INFO:teuthology.orchestra.run.trial024.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2026-02-17T15:12:54.987 INFO:teuthology.orchestra.run.trial024.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2026-02-17T15:12:54.987 INFO:teuthology.orchestra.run.trial024.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2026-02-17T15:12:54.987 INFO:teuthology.orchestra.run.trial024.stdout: python3-threadpoolctl python3-toml python3-waitress python3-wcwidth 2026-02-17T15:12:54.987 INFO:teuthology.orchestra.run.trial024.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2026-02-17T15:12:54.988 INFO:teuthology.orchestra.run.trial024.stdout: python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse socat unzip 2026-02-17T15:12:54.988 INFO:teuthology.orchestra.run.trial024.stdout: xmlstarlet zip 2026-02-17T15:12:55.174 INFO:teuthology.orchestra.run.trial024.stdout:0 upgraded, 108 newly installed, 0 to remove and 51 not upgraded. 2026-02-17T15:12:55.174 INFO:teuthology.orchestra.run.trial024.stdout:Need to get 178 MB of archives. 2026-02-17T15:12:55.175 INFO:teuthology.orchestra.run.trial024.stdout:After this operation, 809 MB of additional disk space will be used. 2026-02-17T15:12:55.175 INFO:teuthology.orchestra.run.trial024.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.2.3-675-g3c96dfb3-1jammy [3597 kB] 2026-02-17T15:12:55.336 INFO:teuthology.orchestra.run.trial024.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-17T15:12:55.425 INFO:teuthology.orchestra.run.trial024.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.2.3-675-g3c96dfb3-1jammy [3257 kB] 2026-02-17T15:12:55.656 INFO:teuthology.orchestra.run.trial024.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.2.3-675-g3c96dfb3-1jammy [979 kB] 2026-02-17T15:12:55.671 INFO:teuthology.orchestra.run.trial024.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-17T15:12:55.682 INFO:teuthology.orchestra.run.trial024.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-17T15:12:55.730 INFO:teuthology.orchestra.run.trial024.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.2.3-675-g3c96dfb3-1jammy [356 kB] 2026-02-17T15:12:55.751 INFO:teuthology.orchestra.run.trial024.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-17T15:12:55.769 INFO:teuthology.orchestra.run.trial024.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.2.3-675-g3c96dfb3-1jammy [32.9 kB] 2026-02-17T15:12:55.873 INFO:teuthology.orchestra.run.trial024.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.2.3-675-g3c96dfb3-1jammy [184 kB] 2026-02-17T15:12:55.932 INFO:teuthology.orchestra.run.trial024.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.2.3-675-g3c96dfb3-1jammy [70.1 kB] 2026-02-17T15:12:55.940 INFO:teuthology.orchestra.run.trial024.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.2.3-675-g3c96dfb3-1jammy [336 kB] 2026-02-17T15:12:55.943 INFO:teuthology.orchestra.run.trial024.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.2.3-675-g3c96dfb3-1jammy [6935 kB] 2026-02-17T15:12:55.952 INFO:teuthology.orchestra.run.trial024.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-17T15:12:55.963 INFO:teuthology.orchestra.run.trial024.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-17T15:12:55.989 INFO:teuthology.orchestra.run.trial024.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-17T15:12:55.997 INFO:teuthology.orchestra.run.trial024.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-17T15:12:55.999 INFO:teuthology.orchestra.run.trial024.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-17T15:12:56.000 INFO:teuthology.orchestra.run.trial024.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-17T15:12:56.001 INFO:teuthology.orchestra.run.trial024.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-17T15:12:56.018 INFO:teuthology.orchestra.run.trial024.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2026-02-17T15:12:56.021 INFO:teuthology.orchestra.run.trial024.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2026-02-17T15:12:56.025 INFO:teuthology.orchestra.run.trial024.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2026-02-17T15:12:56.094 INFO:teuthology.orchestra.run.trial024.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5034 B] 2026-02-17T15:12:56.094 INFO:teuthology.orchestra.run.trial024.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2026-02-17T15:12:56.095 INFO:teuthology.orchestra.run.trial024.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2026-02-17T15:12:56.097 INFO:teuthology.orchestra.run.trial024.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2026-02-17T15:12:56.099 INFO:teuthology.orchestra.run.trial024.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-17T15:12:56.099 INFO:teuthology.orchestra.run.trial024.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-17T15:12:56.100 INFO:teuthology.orchestra.run.trial024.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-17T15:12:56.100 INFO:teuthology.orchestra.run.trial024.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-17T15:12:56.132 INFO:teuthology.orchestra.run.trial024.stdout:Get:32 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.2.3-675-g3c96dfb3-1jammy [112 kB] 2026-02-17T15:12:56.168 INFO:teuthology.orchestra.run.trial024.stdout:Get:33 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.2.3-675-g3c96dfb3-1jammy [470 kB] 2026-02-17T15:12:56.170 INFO:teuthology.orchestra.run.trial024.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-17T15:12:56.171 INFO:teuthology.orchestra.run.trial024.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-17T15:12:56.171 INFO:teuthology.orchestra.run.trial024.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-17T15:12:56.171 INFO:teuthology.orchestra.run.trial024.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-17T15:12:56.172 INFO:teuthology.orchestra.run.trial024.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-17T15:12:56.215 INFO:teuthology.orchestra.run.trial024.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.2.3-675-g3c96dfb3-1jammy [26.5 MB] 2026-02-17T15:12:56.242 INFO:teuthology.orchestra.run.trial024.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-17T15:12:56.244 INFO:teuthology.orchestra.run.trial024.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-17T15:12:56.245 INFO:teuthology.orchestra.run.trial024.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2026-02-17T15:12:56.245 INFO:teuthology.orchestra.run.trial024.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2026-02-17T15:12:56.246 INFO:teuthology.orchestra.run.trial024.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2026-02-17T15:12:56.314 INFO:teuthology.orchestra.run.trial024.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7320 B] 2026-02-17T15:12:56.315 INFO:teuthology.orchestra.run.trial024.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-17T15:12:56.315 INFO:teuthology.orchestra.run.trial024.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-waitress all 1.4.4-1.1ubuntu1.1 [47.0 kB] 2026-02-17T15:12:56.316 INFO:teuthology.orchestra.run.trial024.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2026-02-17T15:12:56.316 INFO:teuthology.orchestra.run.trial024.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2026-02-17T15:12:56.386 INFO:teuthology.orchestra.run.trial024.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4892 B] 2026-02-17T15:12:56.387 INFO:teuthology.orchestra.run.trial024.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2026-02-17T15:12:56.387 INFO:teuthology.orchestra.run.trial024.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2026-02-17T15:12:56.387 INFO:teuthology.orchestra.run.trial024.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2026-02-17T15:12:56.388 INFO:teuthology.orchestra.run.trial024.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.3 [181 kB] 2026-02-17T15:12:56.458 INFO:teuthology.orchestra.run.trial024.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-17T15:12:56.459 INFO:teuthology.orchestra.run.trial024.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-17T15:12:56.461 INFO:teuthology.orchestra.run.trial024.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-17T15:12:56.462 INFO:teuthology.orchestra.run.trial024.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-17T15:12:56.462 INFO:teuthology.orchestra.run.trial024.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-17T15:12:56.530 INFO:teuthology.orchestra.run.trial024.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-17T15:12:56.533 INFO:teuthology.orchestra.run.trial024.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-17T15:12:56.533 INFO:teuthology.orchestra.run.trial024.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-17T15:12:56.553 INFO:teuthology.orchestra.run.trial024.stdout:Get:63 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-17T15:12:56.553 INFO:teuthology.orchestra.run.trial024.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-17T15:12:56.602 INFO:teuthology.orchestra.run.trial024.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-17T15:12:56.603 INFO:teuthology.orchestra.run.trial024.stdout:Get:66 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-17T15:12:56.603 INFO:teuthology.orchestra.run.trial024.stdout:Get:67 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-17T15:12:56.604 INFO:teuthology.orchestra.run.trial024.stdout:Get:68 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-17T15:12:56.607 INFO:teuthology.orchestra.run.trial024.stdout:Get:69 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-17T15:12:56.674 INFO:teuthology.orchestra.run.trial024.stdout:Get:70 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-17T15:12:56.676 INFO:teuthology.orchestra.run.trial024.stdout:Get:71 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-17T15:12:56.676 INFO:teuthology.orchestra.run.trial024.stdout:Get:72 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-17T15:12:56.680 INFO:teuthology.orchestra.run.trial024.stdout:Get:73 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-17T15:12:56.682 INFO:teuthology.orchestra.run.trial024.stdout:Get:74 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2026-02-17T15:12:56.746 INFO:teuthology.orchestra.run.trial024.stdout:Get:75 https://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2026-02-17T15:12:56.747 INFO:teuthology.orchestra.run.trial024.stdout:Get:76 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-17T15:12:56.752 INFO:teuthology.orchestra.run.trial024.stdout:Get:77 https://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2026-02-17T15:12:56.752 INFO:teuthology.orchestra.run.trial024.stdout:Get:78 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-17T15:12:56.755 INFO:teuthology.orchestra.run.trial024.stdout:Get:79 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-17T15:12:56.819 INFO:teuthology.orchestra.run.trial024.stdout:Get:80 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2026-02-17T15:12:56.819 INFO:teuthology.orchestra.run.trial024.stdout:Get:81 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-17T15:12:56.819 INFO:teuthology.orchestra.run.trial024.stdout:Get:82 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-17T15:12:56.821 INFO:teuthology.orchestra.run.trial024.stdout:Get:83 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-17T15:12:56.822 INFO:teuthology.orchestra.run.trial024.stdout:Get:84 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-17T15:12:56.891 INFO:teuthology.orchestra.run.trial024.stdout:Get:85 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2026-02-17T15:12:56.891 INFO:teuthology.orchestra.run.trial024.stdout:Get:86 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-17T15:12:56.891 INFO:teuthology.orchestra.run.trial024.stdout:Get:87 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-17T15:12:56.893 INFO:teuthology.orchestra.run.trial024.stdout:Get:88 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-17T15:12:56.894 INFO:teuthology.orchestra.run.trial024.stdout:Get:89 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-17T15:12:56.905 INFO:teuthology.orchestra.run.trial024.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.2.3-675-g3c96dfb3-1jammy [5178 kB] 2026-02-17T15:12:56.987 INFO:teuthology.orchestra.run.trial024.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.2.3-675-g3c96dfb3-1jammy [248 kB] 2026-02-17T15:12:56.998 INFO:teuthology.orchestra.run.trial024.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.2.3-675-g3c96dfb3-1jammy [124 kB] 2026-02-17T15:12:57.020 INFO:teuthology.orchestra.run.trial024.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.2.3-675-g3c96dfb3-1jammy [1081 kB] 2026-02-17T15:12:57.261 INFO:teuthology.orchestra.run.trial024.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.2.3-675-g3c96dfb3-1jammy [6239 kB] 2026-02-17T15:12:57.574 INFO:teuthology.orchestra.run.trial024.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.2.3-675-g3c96dfb3-1jammy [23.0 MB] 2026-02-17T15:12:58.090 INFO:teuthology.orchestra.run.trial024.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.2.3-675-g3c96dfb3-1jammy [14.2 kB] 2026-02-17T15:12:58.090 INFO:teuthology.orchestra.run.trial024.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.2.3-675-g3c96dfb3-1jammy [1173 kB] 2026-02-17T15:12:58.148 INFO:teuthology.orchestra.run.trial024.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.2.3-675-g3c96dfb3-1jammy [2503 kB] 2026-02-17T15:12:58.420 INFO:teuthology.orchestra.run.trial024.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.2.3-675-g3c96dfb3-1jammy [798 kB] 2026-02-17T15:12:58.433 INFO:teuthology.orchestra.run.trial024.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.2.3-675-g3c96dfb3-1jammy [157 kB] 2026-02-17T15:12:58.482 INFO:teuthology.orchestra.run.trial024.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.2.3-675-g3c96dfb3-1jammy [2396 kB] 2026-02-17T15:12:58.580 INFO:teuthology.orchestra.run.trial024.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.2.3-675-g3c96dfb3-1jammy [8625 kB] 2026-02-17T15:12:58.991 INFO:teuthology.orchestra.run.trial024.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.2.3-675-g3c96dfb3-1jammy [14.3 kB] 2026-02-17T15:12:59.021 INFO:teuthology.orchestra.run.trial024.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.2.3-675-g3c96dfb3-1jammy [52.1 MB] 2026-02-17T15:13:00.271 INFO:teuthology.orchestra.run.trial024.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.2.3-675-g3c96dfb3-1jammy [135 kB] 2026-02-17T15:13:00.284 INFO:teuthology.orchestra.run.trial024.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.2.3-675-g3c96dfb3-1jammy [41.0 kB] 2026-02-17T15:13:00.307 INFO:teuthology.orchestra.run.trial024.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.2.3-675-g3c96dfb3-1jammy [13.7 MB] 2026-02-17T15:13:00.889 INFO:teuthology.orchestra.run.trial024.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/wip-bharath20-testing-2026-02-13-0906-squid/3c96dfb3f50b3d02598a82af7104764bfdcd3919/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.2.3-675-g3c96dfb3-1jammy [92.1 kB] 2026-02-17T15:13:01.297 INFO:teuthology.orchestra.run.trial024.stdout:Fetched 178 MB in 6s (29.6 MB/s) 2026-02-17T15:13:01.313 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-17T15:13:01.468 INFO:teuthology.orchestra.run.trial024.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167438 files and directories currently installed.) 2026-02-17T15:13:01.470 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-17T15:13:01.471 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-17T15:13:01.493 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-17T15:13:01.505 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-17T15:13:01.506 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-17T15:13:01.523 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-17T15:13:01.535 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-17T15:13:01.536 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-17T15:13:01.553 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-17T15:13:01.561 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-17T15:13:01.567 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-17T15:13:01.598 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-17T15:13:01.606 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-17T15:13:01.607 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-17T15:13:01.623 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-17T15:13:01.631 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-17T15:13:01.631 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-17T15:13:01.651 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-17T15:13:01.659 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-17T15:13:01.659 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-17T15:13:01.675 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package librados2. 2026-02-17T15:13:01.683 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../007-librados2_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:01.683 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking librados2 (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:01.723 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libnbd0. 2026-02-17T15:13:01.731 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2026-02-17T15:13:01.731 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-17T15:13:01.746 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package librbd1. 2026-02-17T15:13:01.754 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../009-librbd1_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:01.754 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking librbd1 (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:01.795 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libcephfs2. 2026-02-17T15:13:01.803 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../010-libcephfs2_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:01.804 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libcephfs2 (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:01.824 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-rados. 2026-02-17T15:13:01.832 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../011-python3-rados_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:01.832 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-rados (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:01.849 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-17T15:13:01.857 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.2.3-675-g3c96dfb3-1jammy_all.deb ... 2026-02-17T15:13:01.857 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-ceph-argparse (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:01.872 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-cephfs. 2026-02-17T15:13:01.880 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../013-python3-cephfs_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:01.880 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-cephfs (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:01.895 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-17T15:13:01.903 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../014-python3-ceph-common_19.2.3-675-g3c96dfb3-1jammy_all.deb ... 2026-02-17T15:13:01.903 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-ceph-common (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:01.921 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-17T15:13:01.929 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-17T15:13:01.929 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-17T15:13:01.945 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-prettytable. 2026-02-17T15:13:01.953 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2026-02-17T15:13:01.953 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-17T15:13:01.968 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-rbd. 2026-02-17T15:13:01.976 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../017-python3-rbd_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:01.976 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-rbd (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:01.993 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-17T15:13:02.001 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-17T15:13:02.001 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-17T15:13:02.020 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libreadline-dev:amd64. 2026-02-17T15:13:02.028 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2026-02-17T15:13:02.028 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2026-02-17T15:13:02.044 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2026-02-17T15:13:02.052 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2026-02-17T15:13:02.053 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-17T15:13:02.070 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package lua5.1. 2026-02-17T15:13:02.078 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2026-02-17T15:13:02.079 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2026-02-17T15:13:02.097 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package lua-any. 2026-02-17T15:13:02.105 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2026-02-17T15:13:02.105 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking lua-any (27ubuntu1) ... 2026-02-17T15:13:02.120 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package zip. 2026-02-17T15:13:02.128 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2026-02-17T15:13:02.128 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking zip (3.0-12build2) ... 2026-02-17T15:13:02.143 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package unzip. 2026-02-17T15:13:02.151 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2026-02-17T15:13:02.152 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2026-02-17T15:13:02.170 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package luarocks. 2026-02-17T15:13:02.178 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2026-02-17T15:13:02.179 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2026-02-17T15:13:02.216 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package librgw2. 2026-02-17T15:13:02.224 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../026-librgw2_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:02.224 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking librgw2 (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:02.296 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-rgw. 2026-02-17T15:13:02.304 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../027-python3-rgw_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:02.305 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-rgw (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:02.320 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-17T15:13:02.327 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../028-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-17T15:13:02.328 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-17T15:13:02.343 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libradosstriper1. 2026-02-17T15:13:02.351 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../029-libradosstriper1_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:02.351 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libradosstriper1 (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:02.370 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph-common. 2026-02-17T15:13:02.379 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../030-ceph-common_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:02.379 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph-common (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:02.585 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph-base. 2026-02-17T15:13:02.593 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../031-ceph-base_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:02.598 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph-base (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:02.660 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-17T15:13:02.668 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-17T15:13:02.669 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-17T15:13:02.682 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-cheroot. 2026-02-17T15:13:02.690 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-17T15:13:02.691 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-17T15:13:02.707 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-17T15:13:02.715 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-17T15:13:02.716 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-17T15:13:02.731 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-17T15:13:02.739 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-17T15:13:02.739 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-17T15:13:02.754 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-17T15:13:02.761 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-17T15:13:02.762 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-17T15:13:02.776 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-tempora. 2026-02-17T15:13:02.784 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2026-02-17T15:13:02.785 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-17T15:13:02.799 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-portend. 2026-02-17T15:13:02.807 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2026-02-17T15:13:02.807 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-17T15:13:02.822 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-17T15:13:02.830 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-17T15:13:02.830 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-17T15:13:02.844 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-17T15:13:02.852 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-17T15:13:02.857 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-17T15:13:02.875 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-natsort. 2026-02-17T15:13:02.883 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2026-02-17T15:13:02.883 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-17T15:13:02.899 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-logutils. 2026-02-17T15:13:02.907 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2026-02-17T15:13:02.907 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-logutils (0.3.3-8) ... 2026-02-17T15:13:02.922 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-mako. 2026-02-17T15:13:02.930 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2026-02-17T15:13:02.930 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-17T15:13:02.950 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-simplegeneric. 2026-02-17T15:13:02.957 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2026-02-17T15:13:02.958 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2026-02-17T15:13:02.972 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-singledispatch. 2026-02-17T15:13:02.980 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2026-02-17T15:13:02.981 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2026-02-17T15:13:02.994 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-webob. 2026-02-17T15:13:03.002 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-17T15:13:03.002 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-17T15:13:03.018 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-waitress. 2026-02-17T15:13:03.026 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1.1_all.deb ... 2026-02-17T15:13:03.028 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-17T15:13:03.043 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-tempita. 2026-02-17T15:13:03.051 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2026-02-17T15:13:03.052 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2026-02-17T15:13:03.065 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-paste. 2026-02-17T15:13:03.073 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2026-02-17T15:13:03.074 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2026-02-17T15:13:03.108 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2026-02-17T15:13:03.116 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2026-02-17T15:13:03.116 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2026-02-17T15:13:03.130 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-pastedeploy. 2026-02-17T15:13:03.138 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2026-02-17T15:13:03.138 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2026-02-17T15:13:03.154 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-webtest. 2026-02-17T15:13:03.162 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2026-02-17T15:13:03.162 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-webtest (2.0.35-1) ... 2026-02-17T15:13:03.177 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-pecan. 2026-02-17T15:13:03.185 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2026-02-17T15:13:03.186 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2026-02-17T15:13:03.211 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-werkzeug. 2026-02-17T15:13:03.219 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.3_all.deb ... 2026-02-17T15:13:03.220 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-17T15:13:03.239 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-17T15:13:03.247 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_19.2.3-675-g3c96dfb3-1jammy_all.deb ... 2026-02-17T15:13:03.247 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph-mgr-modules-core (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:03.275 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-17T15:13:03.283 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:03.283 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libsqlite3-mod-ceph (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:03.298 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph-mgr. 2026-02-17T15:13:03.306 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../057-ceph-mgr_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:03.306 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph-mgr (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:03.329 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph-mon. 2026-02-17T15:13:03.337 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../058-ceph-mon_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:03.337 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph-mon (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:03.395 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-17T15:13:03.403 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../059-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-17T15:13:03.403 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-17T15:13:03.419 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph-osd. 2026-02-17T15:13:03.427 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../060-ceph-osd_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:03.427 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph-osd (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:03.584 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph. 2026-02-17T15:13:03.592 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../061-ceph_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:03.592 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:03.606 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph-fuse. 2026-02-17T15:13:03.615 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../062-ceph-fuse_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:03.615 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph-fuse (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:03.638 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph-mds. 2026-02-17T15:13:03.646 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../063-ceph-mds_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:03.647 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph-mds (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:03.687 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package cephadm. 2026-02-17T15:13:03.695 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../064-cephadm_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:03.695 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking cephadm (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:03.711 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-17T15:13:03.719 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-17T15:13:03.720 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-17T15:13:03.741 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-17T15:13:03.749 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../066-ceph-mgr-cephadm_19.2.3-675-g3c96dfb3-1jammy_all.deb ... 2026-02-17T15:13:03.750 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph-mgr-cephadm (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:03.770 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-17T15:13:03.778 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../067-python3-repoze.lru_0.7-2_all.deb ... 2026-02-17T15:13:03.778 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-17T15:13:03.794 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-routes. 2026-02-17T15:13:03.802 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../068-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-17T15:13:03.802 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-17T15:13:03.822 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-17T15:13:03.830 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../069-ceph-mgr-dashboard_19.2.3-675-g3c96dfb3-1jammy_all.deb ... 2026-02-17T15:13:03.830 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph-mgr-dashboard (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:04.032 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-17T15:13:04.040 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../070-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-17T15:13:04.041 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-17T15:13:04.079 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-joblib. 2026-02-17T15:13:04.087 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../071-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-17T15:13:04.087 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-17T15:13:04.114 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-17T15:13:04.123 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../072-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-17T15:13:04.123 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-17T15:13:04.138 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-sklearn. 2026-02-17T15:13:04.146 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../073-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-17T15:13:04.147 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-17T15:13:04.227 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-17T15:13:04.236 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../074-ceph-mgr-diskprediction-local_19.2.3-675-g3c96dfb3-1jammy_all.deb ... 2026-02-17T15:13:04.236 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph-mgr-diskprediction-local (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:04.401 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-cachetools. 2026-02-17T15:13:04.409 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../075-python3-cachetools_5.0.0-1_all.deb ... 2026-02-17T15:13:04.410 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-17T15:13:04.424 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-rsa. 2026-02-17T15:13:04.432 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../076-python3-rsa_4.8-1_all.deb ... 2026-02-17T15:13:04.433 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-17T15:13:04.449 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-google-auth. 2026-02-17T15:13:04.458 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../077-python3-google-auth_1.5.1-3_all.deb ... 2026-02-17T15:13:04.458 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-17T15:13:04.476 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-17T15:13:04.484 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-17T15:13:04.484 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-17T15:13:04.500 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-websocket. 2026-02-17T15:13:04.509 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../079-python3-websocket_1.2.3-1_all.deb ... 2026-02-17T15:13:04.509 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-17T15:13:04.526 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-17T15:13:04.534 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../080-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-17T15:13:04.534 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-17T15:13:04.630 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-17T15:13:04.639 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_19.2.3-675-g3c96dfb3-1jammy_all.deb ... 2026-02-17T15:13:04.639 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph-mgr-k8sevents (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:04.653 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-17T15:13:04.661 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../082-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-17T15:13:04.668 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-17T15:13:04.683 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-17T15:13:04.692 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../083-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-17T15:13:04.692 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-17T15:13:04.707 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package jq. 2026-02-17T15:13:04.715 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../084-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-17T15:13:04.715 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-17T15:13:04.729 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package socat. 2026-02-17T15:13:04.738 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../085-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-17T15:13:04.738 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-17T15:13:04.757 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package xmlstarlet. 2026-02-17T15:13:04.765 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-17T15:13:04.766 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-17T15:13:04.802 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph-test. 2026-02-17T15:13:04.810 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../087-ceph-test_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:04.810 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph-test (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:05.271 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package ceph-volume. 2026-02-17T15:13:05.280 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../088-ceph-volume_19.2.3-675-g3c96dfb3-1jammy_all.deb ... 2026-02-17T15:13:05.280 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking ceph-volume (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:05.302 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-17T15:13:05.310 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../089-libcephfs-dev_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:05.311 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking libcephfs-dev (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:05.326 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package lua-socket:amd64. 2026-02-17T15:13:05.335 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../090-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2026-02-17T15:13:05.335 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-17T15:13:05.356 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package lua-sec:amd64. 2026-02-17T15:13:05.364 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../091-lua-sec_1.0.2-1_amd64.deb ... 2026-02-17T15:13:05.364 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2026-02-17T15:13:05.381 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package nvme-cli. 2026-02-17T15:13:05.389 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../092-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-17T15:13:05.389 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-17T15:13:05.419 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package pkg-config. 2026-02-17T15:13:05.428 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../093-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2026-02-17T15:13:05.428 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2026-02-17T15:13:05.444 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-17T15:13:05.453 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../094-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-17T15:13:05.453 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-17T15:13:05.483 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-17T15:13:05.491 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../095-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-17T15:13:05.491 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-17T15:13:05.506 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-pastescript. 2026-02-17T15:13:05.514 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../096-python3-pastescript_2.0.2-4_all.deb ... 2026-02-17T15:13:05.514 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2026-02-17T15:13:05.532 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-pluggy. 2026-02-17T15:13:05.540 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../097-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-17T15:13:05.541 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-17T15:13:05.557 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-psutil. 2026-02-17T15:13:05.565 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../098-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-17T15:13:05.565 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-17T15:13:05.583 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-py. 2026-02-17T15:13:05.591 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../099-python3-py_1.10.0-1_all.deb ... 2026-02-17T15:13:05.592 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-17T15:13:05.613 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-pygments. 2026-02-17T15:13:05.621 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../100-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-17T15:13:05.621 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-17T15:13:05.662 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-pyinotify. 2026-02-17T15:13:05.671 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../101-python3-pyinotify_0.9.6-1.3_all.deb ... 2026-02-17T15:13:05.671 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2026-02-17T15:13:05.686 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-toml. 2026-02-17T15:13:05.694 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../102-python3-toml_0.10.2-1_all.deb ... 2026-02-17T15:13:05.694 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-17T15:13:05.709 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-pytest. 2026-02-17T15:13:05.717 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../103-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-17T15:13:05.718 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-17T15:13:05.740 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package python3-simplejson. 2026-02-17T15:13:05.748 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../104-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-17T15:13:05.749 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-17T15:13:05.766 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-17T15:13:05.774 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../105-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-17T15:13:05.775 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-17T15:13:05.841 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package radosgw. 2026-02-17T15:13:05.850 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../106-radosgw_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:05.850 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking radosgw (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:05.963 INFO:teuthology.orchestra.run.trial024.stdout:Selecting previously unselected package rbd-fuse. 2026-02-17T15:13:05.971 INFO:teuthology.orchestra.run.trial024.stdout:Preparing to unpack .../107-rbd-fuse_19.2.3-675-g3c96dfb3-1jammy_amd64.deb ... 2026-02-17T15:13:05.972 INFO:teuthology.orchestra.run.trial024.stdout:Unpacking rbd-fuse (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:06.001 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-17T15:13:06.066 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-17T15:13:06.067 INFO:teuthology.orchestra.run.trial024.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-17T15:13:06.140 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-17T15:13:06.410 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-17T15:13:06.763 INFO:teuthology.orchestra.run.trial024.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-17T15:13:06.773 INFO:teuthology.orchestra.run.trial024.stdout:Setting up cephadm (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:06.818 INFO:teuthology.orchestra.run.trial024.stdout:Adding system user cephadm....done 2026-02-17T15:13:06.829 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1.1) ... 2026-02-17T15:13:06.901 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-17T15:13:06.974 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-17T15:13:06.975 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-17T15:13:07.046 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-17T15:13:07.121 INFO:teuthology.orchestra.run.trial024.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-17T15:13:07.122 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-17T15:13:07.217 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-17T15:13:07.341 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-17T15:13:07.417 INFO:teuthology.orchestra.run.trial024.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2026-02-17T15:13:07.427 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2026-02-17T15:13:07.503 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-17T15:13:07.576 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-ceph-argparse (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:07.644 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-17T15:13:07.645 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-17T15:13:07.646 INFO:teuthology.orchestra.run.trial024.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2026-02-17T15:13:07.646 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2026-02-17T15:13:07.647 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-17T15:13:07.648 INFO:teuthology.orchestra.run.trial024.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2026-02-17T15:13:07.665 INFO:teuthology.orchestra.run.trial024.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2026-02-17T15:13:07.667 INFO:teuthology.orchestra.run.trial024.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2026-02-17T15:13:07.668 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-17T15:13:07.669 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-17T15:13:07.788 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-17T15:13:07.864 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-17T15:13:07.941 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-17T15:13:08.030 INFO:teuthology.orchestra.run.trial024.stdout:Setting up zip (3.0-12build2) ... 2026-02-17T15:13:08.031 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-17T15:13:08.280 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2026-02-17T15:13:08.354 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2026-02-17T15:13:08.355 INFO:teuthology.orchestra.run.trial024.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-17T15:13:08.356 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-17T15:13:08.452 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-17T15:13:08.577 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2026-02-17T15:13:08.706 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-17T15:13:08.786 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.3) ... 2026-02-17T15:13:08.890 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-17T15:13:08.963 INFO:teuthology.orchestra.run.trial024.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-17T15:13:08.964 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-ceph-common (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:09.045 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-17T15:13:09.544 INFO:teuthology.orchestra.run.trial024.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2026-02-17T15:13:09.566 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-17T15:13:09.570 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-17T15:13:09.646 INFO:teuthology.orchestra.run.trial024.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-17T15:13:09.647 INFO:teuthology.orchestra.run.trial024.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-17T15:13:09.648 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-17T15:13:09.720 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-17T15:13:09.790 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-17T15:13:09.791 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-17T15:13:09.868 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2026-02-17T15:13:09.943 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-logutils (0.3.3-8) ... 2026-02-17T15:13:10.018 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-17T15:13:10.093 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2026-02-17T15:13:10.166 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-17T15:13:10.244 INFO:teuthology.orchestra.run.trial024.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-17T15:13:10.245 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-17T15:13:10.329 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-17T15:13:10.330 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-17T15:13:10.406 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2026-02-17T15:13:10.487 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-17T15:13:10.572 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-17T15:13:10.644 INFO:teuthology.orchestra.run.trial024.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2026-02-17T15:13:10.645 INFO:teuthology.orchestra.run.trial024.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2026-02-17T15:13:10.646 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-17T15:13:10.647 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-17T15:13:10.780 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2026-02-17T15:13:10.854 INFO:teuthology.orchestra.run.trial024.stdout:Setting up lua-any (27ubuntu1) ... 2026-02-17T15:13:10.855 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-17T15:13:10.927 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-17T15:13:10.928 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-17T15:13:11.010 INFO:teuthology.orchestra.run.trial024.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-17T15:13:11.011 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-webtest (2.0.35-1) ... 2026-02-17T15:13:11.090 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-17T15:13:11.221 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-pastescript (2.0.2-4) ... 2026-02-17T15:13:11.312 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2026-02-17T15:13:11.426 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-17T15:13:11.427 INFO:teuthology.orchestra.run.trial024.stdout:Setting up librados2 (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:11.428 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libsqlite3-mod-ceph (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:11.429 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-17T15:13:11.943 INFO:teuthology.orchestra.run.trial024.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2026-02-17T15:13:11.954 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libcephfs2 (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:11.955 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libradosstriper1 (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:11.956 INFO:teuthology.orchestra.run.trial024.stdout:Setting up librbd1 (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:11.957 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph-mgr-modules-core (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:11.958 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph-fuse (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:12.031 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-17T15:13:12.031 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-17T15:13:12.423 INFO:teuthology.orchestra.run.trial024.stdout:Setting up libcephfs-dev (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:12.424 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-rados (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:12.425 INFO:teuthology.orchestra.run.trial024.stdout:Setting up librgw2 (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:12.426 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-rbd (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:12.427 INFO:teuthology.orchestra.run.trial024.stdout:Setting up rbd-fuse (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:12.427 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-rgw (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:12.428 INFO:teuthology.orchestra.run.trial024.stdout:Setting up python3-cephfs (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:12.429 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph-common (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:12.458 INFO:teuthology.orchestra.run.trial024.stdout:Adding group ceph....done 2026-02-17T15:13:12.492 INFO:teuthology.orchestra.run.trial024.stdout:Adding system user ceph....done 2026-02-17T15:13:12.503 INFO:teuthology.orchestra.run.trial024.stdout:Setting system user ceph properties....done 2026-02-17T15:13:12.509 INFO:teuthology.orchestra.run.trial024.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-17T15:13:12.596 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-17T15:13:12.870 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-17T15:13:13.216 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph-test (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:13.217 INFO:teuthology.orchestra.run.trial024.stdout:Setting up radosgw (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:13.458 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-17T15:13:13.458 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-17T15:13:13.812 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph-base (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:13.932 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-17T15:13:14.293 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph-mds (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:14.353 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-17T15:13:14.353 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-17T15:13:14.693 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph-mgr (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:14.770 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-17T15:13:14.770 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-17T15:13:15.144 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph-osd (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:15.230 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-17T15:13:15.231 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-17T15:13:15.593 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph-mgr-k8sevents (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:15.594 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph-mgr-diskprediction-local (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:15.604 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph-mon (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:15.676 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-17T15:13:15.677 INFO:teuthology.orchestra.run.trial024.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-17T15:13:16.036 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph-mgr-cephadm (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:16.046 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:16.047 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph-mgr-dashboard (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:16.057 INFO:teuthology.orchestra.run.trial024.stdout:Setting up ceph-volume (19.2.3-675-g3c96dfb3-1jammy) ... 2026-02-17T15:13:16.166 INFO:teuthology.orchestra.run.trial024.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2026-02-17T15:13:16.173 INFO:teuthology.orchestra.run.trial024.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.11) ... 2026-02-17T15:13:16.191 INFO:teuthology.orchestra.run.trial024.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-17T15:13:16.279 INFO:teuthology.orchestra.run.trial024.stdout:Processing triggers for install-info (6.8-4build1) ... 2026-02-17T15:13:16.745 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:16.745 INFO:teuthology.orchestra.run.trial024.stdout:Running kernel seems to be up-to-date. 2026-02-17T15:13:16.745 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:16.745 INFO:teuthology.orchestra.run.trial024.stdout:The processor microcode seems to be up-to-date. 2026-02-17T15:13:16.745 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:16.746 INFO:teuthology.orchestra.run.trial024.stdout:No services need to be restarted. 2026-02-17T15:13:16.746 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:16.746 INFO:teuthology.orchestra.run.trial024.stdout:No containers need to be restarted. 2026-02-17T15:13:16.746 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:16.746 INFO:teuthology.orchestra.run.trial024.stdout:No user sessions are running outdated binaries. 2026-02-17T15:13:16.746 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:16.746 INFO:teuthology.orchestra.run.trial024.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-17T15:13:17.375 INFO:teuthology.orchestra.run.trial024.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-17T15:13:17.381 DEBUG:teuthology.parallel:result is None 2026-02-17T15:13:17.382 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3c96dfb3f50b3d02598a82af7104764bfdcd3919 2026-02-17T15:13:17.565 DEBUG:teuthology.orchestra.run.trial024:> dpkg-query -W -f '${Version}' ceph 2026-02-17T15:13:17.576 INFO:teuthology.orchestra.run.trial024.stdout:19.2.3-675-g3c96dfb3-1jammy 2026-02-17T15:13:17.576 INFO:teuthology.packaging:The installed version of ceph is 19.2.3-675-g3c96dfb3-1jammy 2026-02-17T15:13:17.576 INFO:teuthology.task.install:The correct ceph version 19.2.3-675-g3c96dfb3-1jammy is installed. 2026-02-17T15:13:17.579 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-17T15:13:17.579 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:17.579 DEBUG:teuthology.orchestra.run.trial024:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-17T15:13:17.626 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-17T15:13:17.626 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:17.626 DEBUG:teuthology.orchestra.run.trial024:> sudo dd of=/usr/bin/daemon-helper 2026-02-17T15:13:17.678 DEBUG:teuthology.orchestra.run.trial024:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-17T15:13:17.729 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-17T15:13:17.730 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:17.730 DEBUG:teuthology.orchestra.run.trial024:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-17T15:13:17.777 DEBUG:teuthology.orchestra.run.trial024:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-17T15:13:17.825 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-17T15:13:17.825 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:17.826 DEBUG:teuthology.orchestra.run.trial024:> sudo dd of=/usr/bin/stdin-killer 2026-02-17T15:13:17.874 DEBUG:teuthology.orchestra.run.trial024:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-17T15:13:17.925 INFO:teuthology.run_tasks:Running task ceph... 2026-02-17T15:13:18.039 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-17T15:13:18.039 DEBUG:teuthology.orchestra.run.trial024:> sudo chmod 777 /var/log/ceph 2026-02-17T15:13:18.048 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-17T15:13:18.048 DEBUG:teuthology.orchestra.run.trial024:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-17T15:13:18.096 INFO:tasks.ceph:Creating extra log directories... 2026-02-17T15:13:18.096 DEBUG:teuthology.orchestra.run.trial024:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-17T15:13:18.151 INFO:tasks.ceph:Creating ceph cluster ceph... 2026-02-17T15:13:18.151 INFO:tasks.ceph:config {'conf': {'osd': {'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(OBJECT_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SLOW_OPS\\)', 'deep-scrub', 'missing', 'overall HEALTH_', 'repair', 'slow request', 'unfound', '\\(POOL_APP_NOT_ENABLED\\)', 'enough copies available', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-17T15:13:18.152 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/skanta-2026-02-17_04:10:36-rados-wip-bharath20-testing-2026-02-13-0906-squid-distro-default-trial/51568', 'branch': 'wip-bharath20-testing-2026-02-13-0906-squid', 'description': 'rados/singleton/{all/ec-inconsistent-hinfo mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '51568', 'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'trial', 'name': 'skanta-2026-02-17_04:10:36-rados-wip-bharath20-testing-2026-02-13-0906-squid-distro-default-trial', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-bharath20-testing-2026-02-13-0906-squid'}, 'ceph': {'conf': {'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3c96dfb3f50b3d02598a82af7104764bfdcd3919'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3c96dfb3f50b3d02598a82af7104764bfdcd3919'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath20-testing-2026-02-13-0906-squid', 'sha1': '3c96dfb3f50b3d02598a82af7104764bfdcd3919'}}, 'owner': 'scheduled_skanta@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 4581, 'sha1': '3c96dfb3f50b3d02598a82af7104764bfdcd3919', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath20-testing-2026-02-13-0906-squid', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_3c96dfb3f50b3d02598a82af7104764bfdcd3919/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '3c96dfb3f50b3d02598a82af7104764bfdcd3919', 'targets': {'trial024.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBICjGajdMpoZZwL8Eo5R3NDRAZqDvTuH2zl+loisN7xHtg/sOg9BSJM6RNo+floVcuchuWC3YPR/W4lB0Eiwcps='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'create_rbd_pool': False, 'log-ignorelist': ['\\(OBJECT_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(SLOW_OPS\\)', 'deep-scrub', 'missing', 'overall HEALTH_', 'repair', 'slow request', 'unfound', '\\(POOL_APP_NOT_ENABLED\\)', 'enough copies available', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force'], 'flavor': 'default', 'fs': 'xfs', 'sha1': '3c96dfb3f50b3d02598a82af7104764bfdcd3919', 'cluster': 'ceph'}}, {'ec_inconsistent_hinfo': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-17_04:10:36', 'tube': 'trial', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3682'} 2026-02-17T15:13:18.152 DEBUG:teuthology.orchestra.run.trial024:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2026-02-17T15:13:18.189 DEBUG:teuthology.orchestra.run.trial024:> sudo install -d -m0777 -- /var/run/ceph 2026-02-17T15:13:18.237 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:18.238 DEBUG:teuthology.orchestra.run.trial024:> dd if=/scratch_devs of=/dev/stdout 2026-02-17T15:13:18.282 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-17T15:13:18.282 DEBUG:teuthology.orchestra.run.trial024:> stat /dev/vg_nvme/lv_1 2026-02-17T15:13:18.329 INFO:teuthology.orchestra.run.trial024.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-17T15:13:18.329 INFO:teuthology.orchestra.run.trial024.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-17T15:13:18.330 INFO:teuthology.orchestra.run.trial024.stdout:Device: 5h/5d Inode: 898 Links: 1 2026-02-17T15:13:18.330 INFO:teuthology.orchestra.run.trial024.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-17T15:13:18.330 INFO:teuthology.orchestra.run.trial024.stdout:Access: 2026-02-17 15:12:29.577781662 +0000 2026-02-17T15:13:18.330 INFO:teuthology.orchestra.run.trial024.stdout:Modify: 2026-02-17 15:12:29.409787020 +0000 2026-02-17T15:13:18.330 INFO:teuthology.orchestra.run.trial024.stdout:Change: 2026-02-17 15:12:29.409787020 +0000 2026-02-17T15:13:18.330 INFO:teuthology.orchestra.run.trial024.stdout: Birth: - 2026-02-17T15:13:18.330 DEBUG:teuthology.orchestra.run.trial024:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-17T15:13:18.381 INFO:teuthology.orchestra.run.trial024.stderr:1+0 records in 2026-02-17T15:13:18.382 INFO:teuthology.orchestra.run.trial024.stderr:1+0 records out 2026-02-17T15:13:18.382 INFO:teuthology.orchestra.run.trial024.stderr:512 bytes copied, 0.000177457 s, 2.9 MB/s 2026-02-17T15:13:18.383 DEBUG:teuthology.orchestra.run.trial024:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-17T15:13:18.431 DEBUG:teuthology.orchestra.run.trial024:> stat /dev/vg_nvme/lv_2 2026-02-17T15:13:18.477 INFO:teuthology.orchestra.run.trial024.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-17T15:13:18.477 INFO:teuthology.orchestra.run.trial024.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-17T15:13:18.478 INFO:teuthology.orchestra.run.trial024.stdout:Device: 5h/5d Inode: 930 Links: 1 2026-02-17T15:13:18.478 INFO:teuthology.orchestra.run.trial024.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-17T15:13:18.478 INFO:teuthology.orchestra.run.trial024.stdout:Access: 2026-02-17 15:12:29.741776430 +0000 2026-02-17T15:13:18.478 INFO:teuthology.orchestra.run.trial024.stdout:Modify: 2026-02-17 15:12:29.737776561 +0000 2026-02-17T15:13:18.478 INFO:teuthology.orchestra.run.trial024.stdout:Change: 2026-02-17 15:12:29.737776561 +0000 2026-02-17T15:13:18.478 INFO:teuthology.orchestra.run.trial024.stdout: Birth: - 2026-02-17T15:13:18.478 DEBUG:teuthology.orchestra.run.trial024:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-17T15:13:18.528 INFO:teuthology.orchestra.run.trial024.stderr:1+0 records in 2026-02-17T15:13:18.529 INFO:teuthology.orchestra.run.trial024.stderr:1+0 records out 2026-02-17T15:13:18.529 INFO:teuthology.orchestra.run.trial024.stderr:512 bytes copied, 0.000167909 s, 3.0 MB/s 2026-02-17T15:13:18.530 DEBUG:teuthology.orchestra.run.trial024:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-17T15:13:18.579 DEBUG:teuthology.orchestra.run.trial024:> stat /dev/vg_nvme/lv_3 2026-02-17T15:13:18.625 INFO:teuthology.orchestra.run.trial024.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-17T15:13:18.626 INFO:teuthology.orchestra.run.trial024.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-17T15:13:18.626 INFO:teuthology.orchestra.run.trial024.stdout:Device: 5h/5d Inode: 946 Links: 1 2026-02-17T15:13:18.626 INFO:teuthology.orchestra.run.trial024.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-17T15:13:18.626 INFO:teuthology.orchestra.run.trial024.stdout:Access: 2026-02-17 15:12:30.269759590 +0000 2026-02-17T15:13:18.626 INFO:teuthology.orchestra.run.trial024.stdout:Modify: 2026-02-17 15:12:30.073765841 +0000 2026-02-17T15:13:18.626 INFO:teuthology.orchestra.run.trial024.stdout:Change: 2026-02-17 15:12:30.073765841 +0000 2026-02-17T15:13:18.626 INFO:teuthology.orchestra.run.trial024.stdout: Birth: - 2026-02-17T15:13:18.626 DEBUG:teuthology.orchestra.run.trial024:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-17T15:13:18.676 INFO:teuthology.orchestra.run.trial024.stderr:1+0 records in 2026-02-17T15:13:18.676 INFO:teuthology.orchestra.run.trial024.stderr:1+0 records out 2026-02-17T15:13:18.676 INFO:teuthology.orchestra.run.trial024.stderr:512 bytes copied, 0.000172238 s, 3.0 MB/s 2026-02-17T15:13:18.678 DEBUG:teuthology.orchestra.run.trial024:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-17T15:13:18.726 DEBUG:teuthology.orchestra.run.trial024:> stat /dev/vg_nvme/lv_4 2026-02-17T15:13:18.773 INFO:teuthology.orchestra.run.trial024.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-17T15:13:18.773 INFO:teuthology.orchestra.run.trial024.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-17T15:13:18.773 INFO:teuthology.orchestra.run.trial024.stdout:Device: 5h/5d Inode: 971 Links: 1 2026-02-17T15:13:18.774 INFO:teuthology.orchestra.run.trial024.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-17T15:13:18.774 INFO:teuthology.orchestra.run.trial024.stdout:Access: 2026-02-17 15:12:30.613748620 +0000 2026-02-17T15:13:18.774 INFO:teuthology.orchestra.run.trial024.stdout:Modify: 2026-02-17 15:12:30.429754489 +0000 2026-02-17T15:13:18.774 INFO:teuthology.orchestra.run.trial024.stdout:Change: 2026-02-17 15:12:30.429754489 +0000 2026-02-17T15:13:18.774 INFO:teuthology.orchestra.run.trial024.stdout: Birth: - 2026-02-17T15:13:18.774 DEBUG:teuthology.orchestra.run.trial024:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-17T15:13:18.820 INFO:teuthology.orchestra.run.trial024.stderr:1+0 records in 2026-02-17T15:13:18.820 INFO:teuthology.orchestra.run.trial024.stderr:1+0 records out 2026-02-17T15:13:18.821 INFO:teuthology.orchestra.run.trial024.stderr:512 bytes copied, 0.000175192 s, 2.9 MB/s 2026-02-17T15:13:18.822 DEBUG:teuthology.orchestra.run.trial024:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-17T15:13:18.871 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-17T15:13:18.871 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial024.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2026-02-17T15:13:18.871 INFO:tasks.ceph:Generating config... 2026-02-17T15:13:18.873 INFO:tasks.ceph:[osd] osd max pg log entries = 5 2026-02-17T15:13:18.874 INFO:tasks.ceph:[osd] osd min pg log entries = 5 2026-02-17T15:13:18.874 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-17T15:13:18.874 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2026-02-17T15:13:18.874 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2026-02-17T15:13:18.874 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-17T15:13:18.874 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2026-02-17T15:13:18.874 INFO:tasks.ceph:[osd] debug bluefs = 20 2026-02-17T15:13:18.874 INFO:tasks.ceph:[osd] debug bluestore = 20 2026-02-17T15:13:18.874 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-17T15:13:18.874 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-17T15:13:18.875 INFO:tasks.ceph:[osd] debug rocksdb = 10 2026-02-17T15:13:18.875 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-17T15:13:18.875 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-17T15:13:18.875 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-17T15:13:18.875 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2026-02-17T15:13:18.875 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2026-02-17T15:13:18.875 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-17T15:13:18.875 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2026-02-17T15:13:18.875 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2026-02-17T15:13:18.875 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-17T15:13:18.875 INFO:tasks.ceph:[osd] osd op queue = debug_random 2026-02-17T15:13:18.876 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2026-02-17T15:13:18.876 INFO:tasks.ceph:[global] mon election default strategy = 1 2026-02-17T15:13:18.876 INFO:tasks.ceph:[global] ms bind msgr1 = False 2026-02-17T15:13:18.876 INFO:tasks.ceph:[global] ms bind msgr2 = True 2026-02-17T15:13:18.876 INFO:tasks.ceph:[global] ms type = async 2026-02-17T15:13:18.876 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-17T15:13:18.876 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-17T15:13:18.876 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-17T15:13:18.876 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-17T15:13:18.876 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-17T15:13:18.876 INFO:tasks.ceph:[mon] mon scrub interval = 300 2026-02-17T15:13:18.877 INFO:tasks.ceph:Setting up mon.a... 2026-02-17T15:13:18.877 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2026-02-17T15:13:18.930 INFO:teuthology.orchestra.run.trial024.stdout:creating /etc/ceph/ceph.keyring 2026-02-17T15:13:18.934 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2026-02-17T15:13:18.993 DEBUG:teuthology.orchestra.run.trial024:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-17T15:13:19.041 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '10.20.193.24'), ('mon.b', '[v2:10.20.193.24:3301,v1:10.20.193.24:6790]'), ('mon.c', '[v2:10.20.193.24:3302,v1:10.20.193.24:6791]')] 2026-02-17T15:13:19.042 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.24,[v2:10.20.193.24:3301,v1:10.20.193.24:6790],[v2:10.20.193.24:3302,v1:10.20.193.24:6791]', 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd max pg log entries': 5, 'osd min pg log entries': 5, 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2026-02-17T15:13:19.043 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:19.044 DEBUG:teuthology.orchestra.run.trial024:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-17T15:13:19.091 DEBUG:teuthology.orchestra.run.trial024:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.24 --addv b '[v2:10.20.193.24:3301,v1:10.20.193.24:6790]' --addv c '[v2:10.20.193.24:3302,v1:10.20.193.24:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2026-02-17T15:13:19.152 INFO:teuthology.orchestra.run.trial024.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2026-02-17T15:13:19.152 INFO:teuthology.orchestra.run.trial024.stdout:monmaptool: generated fsid ac9bccef-327a-4812-a379-08d174e6a4bc 2026-02-17T15:13:19.152 INFO:teuthology.orchestra.run.trial024.stdout:setting min_mon_release = quincy 2026-02-17T15:13:19.153 INFO:teuthology.orchestra.run.trial024.stdout:epoch 0 2026-02-17T15:13:19.153 INFO:teuthology.orchestra.run.trial024.stdout:fsid ac9bccef-327a-4812-a379-08d174e6a4bc 2026-02-17T15:13:19.153 INFO:teuthology.orchestra.run.trial024.stdout:last_changed 2026-02-17T15:13:19.151203+0000 2026-02-17T15:13:19.153 INFO:teuthology.orchestra.run.trial024.stdout:created 2026-02-17T15:13:19.151203+0000 2026-02-17T15:13:19.153 INFO:teuthology.orchestra.run.trial024.stdout:min_mon_release 17 (quincy) 2026-02-17T15:13:19.153 INFO:teuthology.orchestra.run.trial024.stdout:election_strategy: 1 2026-02-17T15:13:19.153 INFO:teuthology.orchestra.run.trial024.stdout:0: [v2:10.20.193.24:3300/0,v1:10.20.193.24:6789/0] mon.a 2026-02-17T15:13:19.153 INFO:teuthology.orchestra.run.trial024.stdout:1: [v2:10.20.193.24:3301/0,v1:10.20.193.24:6790/0] mon.b 2026-02-17T15:13:19.153 INFO:teuthology.orchestra.run.trial024.stdout:2: [v2:10.20.193.24:3302/0,v1:10.20.193.24:6791/0] mon.c 2026-02-17T15:13:19.153 INFO:teuthology.orchestra.run.trial024.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2026-02-17T15:13:19.155 DEBUG:teuthology.orchestra.run.trial024:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-17T15:13:19.201 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ac9bccef-327a-4812-a379-08d174e6a4bc... 2026-02-17T15:13:19.203 DEBUG:teuthology.orchestra.run.trial024:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2026-02-17T15:13:19.261 INFO:teuthology.orchestra.run.trial024.stdout:[global] 2026-02-17T15:13:19.261 INFO:teuthology.orchestra.run.trial024.stdout: chdir = "" 2026-02-17T15:13:19.261 INFO:teuthology.orchestra.run.trial024.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-17T15:13:19.261 INFO:teuthology.orchestra.run.trial024.stdout: auth supported = cephx 2026-02-17T15:13:19.262 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.262 INFO:teuthology.orchestra.run.trial024.stdout: filestore xattr use omap = true 2026-02-17T15:13:19.262 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.262 INFO:teuthology.orchestra.run.trial024.stdout: mon clock drift allowed = 1.000 2026-02-17T15:13:19.262 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.262 INFO:teuthology.orchestra.run.trial024.stdout: osd crush chooseleaf type = 0 2026-02-17T15:13:19.262 INFO:teuthology.orchestra.run.trial024.stdout: auth debug = true 2026-02-17T15:13:19.262 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.262 INFO:teuthology.orchestra.run.trial024.stdout: ms die on old message = true 2026-02-17T15:13:19.262 INFO:teuthology.orchestra.run.trial024.stdout: ms die on bug = true 2026-02-17T15:13:19.262 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.263 INFO:teuthology.orchestra.run.trial024.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-17T15:13:19.263 INFO:teuthology.orchestra.run.trial024.stdout: mon pg warn max object skew = 0 2026-02-17T15:13:19.263 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.263 INFO:teuthology.orchestra.run.trial024.stdout: # disable pg_autoscaler by default for new pools 2026-02-17T15:13:19.263 INFO:teuthology.orchestra.run.trial024.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-17T15:13:19.263 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.263 INFO:teuthology.orchestra.run.trial024.stdout: osd pool default size = 2 2026-02-17T15:13:19.263 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.263 INFO:teuthology.orchestra.run.trial024.stdout: mon osd allow primary affinity = true 2026-02-17T15:13:19.263 INFO:teuthology.orchestra.run.trial024.stdout: mon osd allow pg remap = true 2026-02-17T15:13:19.263 INFO:teuthology.orchestra.run.trial024.stdout: mon warn on legacy crush tunables = false 2026-02-17T15:13:19.263 INFO:teuthology.orchestra.run.trial024.stdout: mon warn on crush straw calc version zero = false 2026-02-17T15:13:19.264 INFO:teuthology.orchestra.run.trial024.stdout: mon warn on no sortbitwise = false 2026-02-17T15:13:19.264 INFO:teuthology.orchestra.run.trial024.stdout: mon warn on osd down out interval zero = false 2026-02-17T15:13:19.264 INFO:teuthology.orchestra.run.trial024.stdout: mon warn on too few osds = false 2026-02-17T15:13:19.264 INFO:teuthology.orchestra.run.trial024.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-17T15:13:19.264 INFO:teuthology.orchestra.run.trial024.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-17T15:13:19.264 INFO:teuthology.orchestra.run.trial024.stdout: mon_allow_pool_size_one = true 2026-02-17T15:13:19.264 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.264 INFO:teuthology.orchestra.run.trial024.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-17T15:13:19.264 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.264 INFO:teuthology.orchestra.run.trial024.stdout: osd default data pool replay window = 5 2026-02-17T15:13:19.264 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.264 INFO:teuthology.orchestra.run.trial024.stdout: mon allow pool delete = true 2026-02-17T15:13:19.264 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.265 INFO:teuthology.orchestra.run.trial024.stdout: mon cluster log file level = debug 2026-02-17T15:13:19.265 INFO:teuthology.orchestra.run.trial024.stdout: debug asserts on shutdown = true 2026-02-17T15:13:19.265 INFO:teuthology.orchestra.run.trial024.stdout: mon health detail to clog = false 2026-02-17T15:13:19.265 INFO:teuthology.orchestra.run.trial024.stdout: mon host = "10.20.193.24,[v2:10.20.193.24:3301,v1:10.20.193.24:6790],[v2:10.20.193.24:3302,v1:10.20.193.24:6791]" 2026-02-17T15:13:19.265 INFO:teuthology.orchestra.run.trial024.stdout: mon election default strategy = 1 2026-02-17T15:13:19.265 INFO:teuthology.orchestra.run.trial024.stdout: ms bind msgr1 = False 2026-02-17T15:13:19.265 INFO:teuthology.orchestra.run.trial024.stdout: ms bind msgr2 = True 2026-02-17T15:13:19.265 INFO:teuthology.orchestra.run.trial024.stdout: ms type = async 2026-02-17T15:13:19.265 INFO:teuthology.orchestra.run.trial024.stdout: fsid = ac9bccef-327a-4812-a379-08d174e6a4bc 2026-02-17T15:13:19.265 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.265 INFO:teuthology.orchestra.run.trial024.stdout:[osd] 2026-02-17T15:13:19.265 INFO:teuthology.orchestra.run.trial024.stdout: osd journal size = 100 2026-02-17T15:13:19.266 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.266 INFO:teuthology.orchestra.run.trial024.stdout: osd scrub load threshold = 5.0 2026-02-17T15:13:19.266 INFO:teuthology.orchestra.run.trial024.stdout: osd scrub max interval = 600 2026-02-17T15:13:19.266 INFO:teuthology.orchestra.run.trial024.stdout: osd mclock profile = high_recovery_ops 2026-02-17T15:13:19.266 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.266 INFO:teuthology.orchestra.run.trial024.stdout: osd recover clone overlap = true 2026-02-17T15:13:19.266 INFO:teuthology.orchestra.run.trial024.stdout: osd recovery max chunk = 1048576 2026-02-17T15:13:19.266 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.266 INFO:teuthology.orchestra.run.trial024.stdout: osd debug shutdown = true 2026-02-17T15:13:19.266 INFO:teuthology.orchestra.run.trial024.stdout: osd debug op order = true 2026-02-17T15:13:19.266 INFO:teuthology.orchestra.run.trial024.stdout: osd debug verify stray on activate = true 2026-02-17T15:13:19.266 INFO:teuthology.orchestra.run.trial024.stdout: osd debug trim objects = true 2026-02-17T15:13:19.266 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.267 INFO:teuthology.orchestra.run.trial024.stdout: osd open classes on start = true 2026-02-17T15:13:19.267 INFO:teuthology.orchestra.run.trial024.stdout: osd debug pg log writeout = true 2026-02-17T15:13:19.267 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.267 INFO:teuthology.orchestra.run.trial024.stdout: osd deep scrub update digest min age = 30 2026-02-17T15:13:19.267 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.267 INFO:teuthology.orchestra.run.trial024.stdout: osd map max advance = 10 2026-02-17T15:13:19.267 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.267 INFO:teuthology.orchestra.run.trial024.stdout: journal zero on create = true 2026-02-17T15:13:19.267 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.267 INFO:teuthology.orchestra.run.trial024.stdout: filestore ondisk finisher threads = 3 2026-02-17T15:13:19.267 INFO:teuthology.orchestra.run.trial024.stdout: filestore apply finisher threads = 3 2026-02-17T15:13:19.267 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.268 INFO:teuthology.orchestra.run.trial024.stdout: bdev debug aio = true 2026-02-17T15:13:19.268 INFO:teuthology.orchestra.run.trial024.stdout: osd debug misdirected ops = true 2026-02-17T15:13:19.268 INFO:teuthology.orchestra.run.trial024.stdout: osd max pg log entries = 5 2026-02-17T15:13:19.268 INFO:teuthology.orchestra.run.trial024.stdout: osd min pg log entries = 5 2026-02-17T15:13:19.268 INFO:teuthology.orchestra.run.trial024.stdout: bluestore block size = 96636764160 2026-02-17T15:13:19.268 INFO:teuthology.orchestra.run.trial024.stdout: bluestore compression algorithm = zstd 2026-02-17T15:13:19.268 INFO:teuthology.orchestra.run.trial024.stdout: bluestore compression mode = aggressive 2026-02-17T15:13:19.268 INFO:teuthology.orchestra.run.trial024.stdout: bluestore fsck on mount = True 2026-02-17T15:13:19.268 INFO:teuthology.orchestra.run.trial024.stdout: bluestore zero block detection = True 2026-02-17T15:13:19.268 INFO:teuthology.orchestra.run.trial024.stdout: debug bluefs = 20 2026-02-17T15:13:19.268 INFO:teuthology.orchestra.run.trial024.stdout: debug bluestore = 20 2026-02-17T15:13:19.268 INFO:teuthology.orchestra.run.trial024.stdout: debug ms = 1 2026-02-17T15:13:19.268 INFO:teuthology.orchestra.run.trial024.stdout: debug osd = 20 2026-02-17T15:13:19.268 INFO:teuthology.orchestra.run.trial024.stdout: debug rocksdb = 10 2026-02-17T15:13:19.269 INFO:teuthology.orchestra.run.trial024.stdout: mon osd backfillfull_ratio = 0.85 2026-02-17T15:13:19.269 INFO:teuthology.orchestra.run.trial024.stdout: mon osd full ratio = 0.9 2026-02-17T15:13:19.269 INFO:teuthology.orchestra.run.trial024.stdout: mon osd nearfull ratio = 0.8 2026-02-17T15:13:19.269 INFO:teuthology.orchestra.run.trial024.stdout: osd debug verify cached snaps = True 2026-02-17T15:13:19.269 INFO:teuthology.orchestra.run.trial024.stdout: osd debug verify missing on start = True 2026-02-17T15:13:19.269 INFO:teuthology.orchestra.run.trial024.stdout: osd failsafe full ratio = 0.95 2026-02-17T15:13:19.269 INFO:teuthology.orchestra.run.trial024.stdout: osd mclock override recovery settings = True 2026-02-17T15:13:19.269 INFO:teuthology.orchestra.run.trial024.stdout: osd objectstore = bluestore 2026-02-17T15:13:19.269 INFO:teuthology.orchestra.run.trial024.stdout: osd op queue = debug_random 2026-02-17T15:13:19.269 INFO:teuthology.orchestra.run.trial024.stdout: osd op queue cut off = debug_random 2026-02-17T15:13:19.269 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.269 INFO:teuthology.orchestra.run.trial024.stdout:[mgr] 2026-02-17T15:13:19.270 INFO:teuthology.orchestra.run.trial024.stdout: debug ms = 1 2026-02-17T15:13:19.270 INFO:teuthology.orchestra.run.trial024.stdout: debug mgr = 20 2026-02-17T15:13:19.270 INFO:teuthology.orchestra.run.trial024.stdout: debug mon = 20 2026-02-17T15:13:19.270 INFO:teuthology.orchestra.run.trial024.stdout: debug auth = 20 2026-02-17T15:13:19.270 INFO:teuthology.orchestra.run.trial024.stdout: mon reweight min pgs per osd = 4 2026-02-17T15:13:19.270 INFO:teuthology.orchestra.run.trial024.stdout: mon reweight min bytes per osd = 10 2026-02-17T15:13:19.270 INFO:teuthology.orchestra.run.trial024.stdout: mgr/telemetry/nag = false 2026-02-17T15:13:19.270 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.270 INFO:teuthology.orchestra.run.trial024.stdout:[mon] 2026-02-17T15:13:19.270 INFO:teuthology.orchestra.run.trial024.stdout: debug ms = 1 2026-02-17T15:13:19.270 INFO:teuthology.orchestra.run.trial024.stdout: debug mon = 20 2026-02-17T15:13:19.270 INFO:teuthology.orchestra.run.trial024.stdout: debug paxos = 20 2026-02-17T15:13:19.270 INFO:teuthology.orchestra.run.trial024.stdout: debug auth = 20 2026-02-17T15:13:19.271 INFO:teuthology.orchestra.run.trial024.stdout: mon data avail warn = 5 2026-02-17T15:13:19.271 INFO:teuthology.orchestra.run.trial024.stdout: mon mgr mkfs grace = 240 2026-02-17T15:13:19.271 INFO:teuthology.orchestra.run.trial024.stdout: mon reweight min pgs per osd = 4 2026-02-17T15:13:19.271 INFO:teuthology.orchestra.run.trial024.stdout: mon osd reporter subtree level = osd 2026-02-17T15:13:19.271 INFO:teuthology.orchestra.run.trial024.stdout: mon osd prime pg temp = true 2026-02-17T15:13:19.271 INFO:teuthology.orchestra.run.trial024.stdout: mon reweight min bytes per osd = 10 2026-02-17T15:13:19.271 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.271 INFO:teuthology.orchestra.run.trial024.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-17T15:13:19.271 INFO:teuthology.orchestra.run.trial024.stdout: auth mon ticket ttl = 660 # 11m 2026-02-17T15:13:19.271 INFO:teuthology.orchestra.run.trial024.stdout: auth service ticket ttl = 240 # 4m 2026-02-17T15:13:19.271 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.271 INFO:teuthology.orchestra.run.trial024.stdout: # don't complain about insecure global_id in the test suite 2026-02-17T15:13:19.271 INFO:teuthology.orchestra.run.trial024.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-17T15:13:19.272 INFO:teuthology.orchestra.run.trial024.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-17T15:13:19.272 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.272 INFO:teuthology.orchestra.run.trial024.stdout: # 1m isn't quite enough 2026-02-17T15:13:19.272 INFO:teuthology.orchestra.run.trial024.stdout: mon_down_mkfs_grace = 2m 2026-02-17T15:13:19.272 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.272 INFO:teuthology.orchestra.run.trial024.stdout: mon_warn_on_filestore_osds = false 2026-02-17T15:13:19.272 INFO:teuthology.orchestra.run.trial024.stdout: mon scrub interval = 300 2026-02-17T15:13:19.272 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:19.272 INFO:teuthology.orchestra.run.trial024.stdout:[client] 2026-02-17T15:13:19.272 INFO:teuthology.orchestra.run.trial024.stdout: rgw cache enabled = true 2026-02-17T15:13:19.272 INFO:teuthology.orchestra.run.trial024.stdout: rgw enable ops log = true 2026-02-17T15:13:19.272 INFO:teuthology.orchestra.run.trial024.stdout: rgw enable usage log = true 2026-02-17T15:13:19.272 INFO:teuthology.orchestra.run.trial024.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-17T15:13:19.273 INFO:teuthology.orchestra.run.trial024.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-17T15:13:19.273 INFO:teuthology.orchestra.run.trial024.stdout:[mon.a] 2026-02-17T15:13:19.273 INFO:teuthology.orchestra.run.trial024.stdout:[mon.b] 2026-02-17T15:13:19.273 INFO:teuthology.orchestra.run.trial024.stdout:[mon.c] 2026-02-17T15:13:19.274 INFO:tasks.ceph:Creating admin key on mon.a... 2026-02-17T15:13:19.274 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2026-02-17T15:13:19.339 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-17T15:13:19.339 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:19.340 DEBUG:teuthology.orchestra.run.trial024:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2026-02-17T15:13:19.391 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:19.391 DEBUG:teuthology.orchestra.run.trial024:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2026-02-17T15:13:19.442 INFO:tasks.ceph:Sending monmap to node ubuntu@trial024.front.sepia.ceph.com 2026-02-17T15:13:19.442 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:19.442 DEBUG:teuthology.orchestra.run.trial024:> sudo dd of=/etc/ceph/ceph.keyring 2026-02-17T15:13:19.442 DEBUG:teuthology.orchestra.run.trial024:> sudo chmod 0644 /etc/ceph/ceph.keyring 2026-02-17T15:13:19.499 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:19.499 DEBUG:teuthology.orchestra.run.trial024:> dd of=/home/ubuntu/cephtest/ceph.monmap 2026-02-17T15:13:19.546 INFO:tasks.ceph:Setting up mon nodes... 2026-02-17T15:13:19.547 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-17T15:13:19.547 DEBUG:teuthology.orchestra.run.trial024:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2026-02-17T15:13:19.617 INFO:teuthology.orchestra.run.trial024.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2026-02-17T15:13:19.619 INFO:tasks.ceph:Setting up mds nodes... 2026-02-17T15:13:19.620 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-17T15:13:19.620 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-17T15:13:19.620 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial024.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2026-02-17T15:13:19.620 DEBUG:teuthology.orchestra.run.trial024:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2026-02-17T15:13:19.669 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-17T15:13:19.670 INFO:tasks.ceph:role: osd.0 2026-02-17T15:13:19.670 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial024.front.sepia.ceph.com 2026-02-17T15:13:19.670 DEBUG:teuthology.orchestra.run.trial024:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-17T15:13:19.723 INFO:teuthology.orchestra.run.trial024.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-17T15:13:19.723 INFO:teuthology.orchestra.run.trial024.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-17T15:13:19.723 INFO:teuthology.orchestra.run.trial024.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-17T15:13:19.724 INFO:teuthology.orchestra.run.trial024.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-17T15:13:19.724 INFO:teuthology.orchestra.run.trial024.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-17T15:13:19.724 INFO:teuthology.orchestra.run.trial024.stdout: = sunit=0 swidth=0 blks 2026-02-17T15:13:19.724 INFO:teuthology.orchestra.run.trial024.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-17T15:13:19.724 INFO:teuthology.orchestra.run.trial024.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-17T15:13:19.724 INFO:teuthology.orchestra.run.trial024.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-17T15:13:19.725 INFO:teuthology.orchestra.run.trial024.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-17T15:13:20.299 INFO:teuthology.orchestra.run.trial024.stdout:Discarding blocks...Done. 2026-02-17T15:13:20.300 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial024.front.sepia.ceph.com -o noatime 2026-02-17T15:13:20.301 DEBUG:teuthology.orchestra.run.trial024:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2026-02-17T15:13:20.359 DEBUG:teuthology.orchestra.run.trial024:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2026-02-17T15:13:20.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-17T15:13:20.412 INFO:teuthology.orchestra.run.trial024.stderr:sudo: /sbin/restorecon: command not found 2026-02-17T15:13:20.412 DEBUG:teuthology.orchestra.run.trial024:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2026-02-17T15:13:20.462 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-17T15:13:20.462 INFO:tasks.ceph:role: osd.1 2026-02-17T15:13:20.462 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial024.front.sepia.ceph.com 2026-02-17T15:13:20.462 DEBUG:teuthology.orchestra.run.trial024:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-17T15:13:20.516 INFO:teuthology.orchestra.run.trial024.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-17T15:13:20.516 INFO:teuthology.orchestra.run.trial024.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-17T15:13:20.516 INFO:teuthology.orchestra.run.trial024.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-17T15:13:20.516 INFO:teuthology.orchestra.run.trial024.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-17T15:13:20.517 INFO:teuthology.orchestra.run.trial024.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-17T15:13:20.517 INFO:teuthology.orchestra.run.trial024.stdout: = sunit=0 swidth=0 blks 2026-02-17T15:13:20.517 INFO:teuthology.orchestra.run.trial024.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-17T15:13:20.517 INFO:teuthology.orchestra.run.trial024.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-17T15:13:20.517 INFO:teuthology.orchestra.run.trial024.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-17T15:13:20.517 INFO:teuthology.orchestra.run.trial024.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-17T15:13:21.081 INFO:teuthology.orchestra.run.trial024.stdout:Discarding blocks...Done. 2026-02-17T15:13:21.083 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial024.front.sepia.ceph.com -o noatime 2026-02-17T15:13:21.083 DEBUG:teuthology.orchestra.run.trial024:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2026-02-17T15:13:21.143 DEBUG:teuthology.orchestra.run.trial024:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2026-02-17T15:13:21.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-17T15:13:21.192 INFO:teuthology.orchestra.run.trial024.stderr:sudo: /sbin/restorecon: command not found 2026-02-17T15:13:21.192 DEBUG:teuthology.orchestra.run.trial024:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2026-02-17T15:13:21.241 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-17T15:13:21.242 INFO:tasks.ceph:role: osd.2 2026-02-17T15:13:21.242 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial024.front.sepia.ceph.com 2026-02-17T15:13:21.242 DEBUG:teuthology.orchestra.run.trial024:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-17T15:13:21.295 INFO:teuthology.orchestra.run.trial024.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-17T15:13:21.296 INFO:teuthology.orchestra.run.trial024.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-17T15:13:21.296 INFO:teuthology.orchestra.run.trial024.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-17T15:13:21.296 INFO:teuthology.orchestra.run.trial024.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-17T15:13:21.296 INFO:teuthology.orchestra.run.trial024.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-17T15:13:21.296 INFO:teuthology.orchestra.run.trial024.stdout: = sunit=0 swidth=0 blks 2026-02-17T15:13:21.296 INFO:teuthology.orchestra.run.trial024.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-17T15:13:21.296 INFO:teuthology.orchestra.run.trial024.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-17T15:13:21.296 INFO:teuthology.orchestra.run.trial024.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-17T15:13:21.296 INFO:teuthology.orchestra.run.trial024.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-17T15:13:21.858 INFO:teuthology.orchestra.run.trial024.stdout:Discarding blocks...Done. 2026-02-17T15:13:21.860 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial024.front.sepia.ceph.com -o noatime 2026-02-17T15:13:21.860 DEBUG:teuthology.orchestra.run.trial024:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2026-02-17T15:13:21.918 DEBUG:teuthology.orchestra.run.trial024:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2026-02-17T15:13:21.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-17T15:13:21.968 INFO:teuthology.orchestra.run.trial024.stderr:sudo: /sbin/restorecon: command not found 2026-02-17T15:13:21.968 DEBUG:teuthology.orchestra.run.trial024:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2026-02-17T15:13:22.017 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2026-02-17T15:13:22.018 INFO:tasks.ceph:role: osd.3 2026-02-17T15:13:22.018 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@trial024.front.sepia.ceph.com 2026-02-17T15:13:22.018 DEBUG:teuthology.orchestra.run.trial024:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2026-02-17T15:13:22.071 INFO:teuthology.orchestra.run.trial024.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=45875200 blks 2026-02-17T15:13:22.071 INFO:teuthology.orchestra.run.trial024.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-17T15:13:22.071 INFO:teuthology.orchestra.run.trial024.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-17T15:13:22.071 INFO:teuthology.orchestra.run.trial024.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-17T15:13:22.071 INFO:teuthology.orchestra.run.trial024.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-17T15:13:22.071 INFO:teuthology.orchestra.run.trial024.stdout: = sunit=0 swidth=0 blks 2026-02-17T15:13:22.072 INFO:teuthology.orchestra.run.trial024.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-17T15:13:22.072 INFO:teuthology.orchestra.run.trial024.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-17T15:13:22.072 INFO:teuthology.orchestra.run.trial024.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-17T15:13:22.072 INFO:teuthology.orchestra.run.trial024.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-17T15:13:22.634 INFO:teuthology.orchestra.run.trial024.stdout:Discarding blocks...Done. 2026-02-17T15:13:22.635 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@trial024.front.sepia.ceph.com -o noatime 2026-02-17T15:13:22.636 DEBUG:teuthology.orchestra.run.trial024:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2026-02-17T15:13:22.690 DEBUG:teuthology.orchestra.run.trial024:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2026-02-17T15:13:22.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-17T15:13:22.740 INFO:teuthology.orchestra.run.trial024.stderr:sudo: /sbin/restorecon: command not found 2026-02-17T15:13:22.740 DEBUG:teuthology.orchestra.run.trial024:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-17T15:13:22.804 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:22.799+0000 7f3341eeb8c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2026-02-17T15:13:22.805 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:22.799+0000 7f3341eeb8c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2026-02-17T15:13:22.805 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:22.799+0000 7f3341eeb8c0 -1 bdev(0x560ac50a5800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2026-02-17T15:13:22.805 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:22.799+0000 7f3341eeb8c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2026-02-17T15:13:25.906 DEBUG:teuthology.orchestra.run.trial024:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2026-02-17T15:13:25.958 DEBUG:teuthology.orchestra.run.trial024:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-17T15:13:26.022 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:26.019+0000 7fa3ca2df8c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2026-02-17T15:13:26.022 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:26.019+0000 7fa3ca2df8c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2026-02-17T15:13:26.023 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:26.019+0000 7fa3ca2df8c0 -1 bdev(0x5616292d1800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2026-02-17T15:13:26.023 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:26.019+0000 7fa3ca2df8c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2026-02-17T15:13:29.166 DEBUG:teuthology.orchestra.run.trial024:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2026-02-17T15:13:29.214 DEBUG:teuthology.orchestra.run.trial024:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-17T15:13:29.278 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:29.275+0000 7ff58b3d98c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2026-02-17T15:13:29.279 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:29.275+0000 7ff58b3d98c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2026-02-17T15:13:29.279 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:29.275+0000 7ff58b3d98c0 -1 bdev(0x558a8dff3800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2026-02-17T15:13:29.279 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:29.275+0000 7ff58b3d98c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2026-02-17T15:13:32.389 DEBUG:teuthology.orchestra.run.trial024:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2026-02-17T15:13:32.439 DEBUG:teuthology.orchestra.run.trial024:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2026-02-17T15:13:32.502 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:32.495+0000 7faf10fba8c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2026-02-17T15:13:32.502 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:32.499+0000 7faf10fba8c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2026-02-17T15:13:32.503 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:32.499+0000 7faf10fba8c0 -1 bdev(0x56317ef5d800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2026-02-17T15:13:32.503 INFO:teuthology.orchestra.run.trial024.stderr:2026-02-17T15:13:32.499+0000 7faf10fba8c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2026-02-17T15:13:35.621 DEBUG:teuthology.orchestra.run.trial024:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2026-02-17T15:13:35.671 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-17T15:13:35.671 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:35.671 DEBUG:teuthology.orchestra.run.trial024:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2026-02-17T15:13:35.723 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:35.723 DEBUG:teuthology.orchestra.run.trial024:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2026-02-17T15:13:35.774 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:35.775 DEBUG:teuthology.orchestra.run.trial024:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2026-02-17T15:13:35.825 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:35.826 DEBUG:teuthology.orchestra.run.trial024:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2026-02-17T15:13:35.875 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:35.875 DEBUG:teuthology.orchestra.run.trial024:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2026-02-17T15:13:35.927 INFO:tasks.ceph:Adding keys to all mons... 2026-02-17T15:13:35.927 DEBUG:teuthology.orchestra.run.trial024:> sudo tee -a /etc/ceph/ceph.keyring 2026-02-17T15:13:35.976 INFO:teuthology.orchestra.run.trial024.stdout:[mgr.x] 2026-02-17T15:13:35.976 INFO:teuthology.orchestra.run.trial024.stdout: key = AQCPhZRptAi5JBAA7Hn8BhKV9TTdJvz2Ke96FQ== 2026-02-17T15:13:35.977 INFO:teuthology.orchestra.run.trial024.stdout:[osd.0] 2026-02-17T15:13:35.977 INFO:teuthology.orchestra.run.trial024.stdout: key = AQCShZRpK2zgLxAA0Mtd7cFhE8glH0mx+HMRFg== 2026-02-17T15:13:35.977 INFO:teuthology.orchestra.run.trial024.stdout:[osd.1] 2026-02-17T15:13:35.977 INFO:teuthology.orchestra.run.trial024.stdout: key = AQCWhZRpsQBAARAAi58qu45N8TX/ffxKXJxSoQ== 2026-02-17T15:13:35.977 INFO:teuthology.orchestra.run.trial024.stdout:[osd.2] 2026-02-17T15:13:35.977 INFO:teuthology.orchestra.run.trial024.stdout: key = AQCZhZRpLE6MEBAAD8InMeq5GQIPjgAbL2X+8Q== 2026-02-17T15:13:35.977 INFO:teuthology.orchestra.run.trial024.stdout:[osd.3] 2026-02-17T15:13:35.977 INFO:teuthology.orchestra.run.trial024.stdout: key = AQCchZRpxozZHRAADBcEfEtLHUnP5lbJ0IE9wg== 2026-02-17T15:13:35.978 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-17T15:13:36.043 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-17T15:13:36.106 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-17T15:13:36.170 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-17T15:13:36.234 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-17T15:13:36.298 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-17T15:13:36.298 DEBUG:teuthology.orchestra.run.trial024:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2026-02-17T15:13:36.350 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-17T15:13:36.420 DEBUG:teuthology.orchestra.run.trial024:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2026-02-17T15:13:36.474 DEBUG:teuthology.orchestra.run.trial024:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2026-02-17T15:13:36.530 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-17T15:13:36.600 DEBUG:teuthology.orchestra.run.trial024:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2026-02-17T15:13:36.654 DEBUG:teuthology.orchestra.run.trial024:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2026-02-17T15:13:36.710 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2026-02-17T15:13:36.775 DEBUG:teuthology.orchestra.run.trial024:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2026-02-17T15:13:36.826 DEBUG:teuthology.orchestra.run.trial024:> rm -- /home/ubuntu/cephtest/ceph.monmap 2026-02-17T15:13:36.873 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2026-02-17T15:13:36.874 INFO:tasks.ceph.mon.a:Restarting daemon 2026-02-17T15:13:36.874 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2026-02-17T15:13:36.915 INFO:tasks.ceph.mon.a:Started 2026-02-17T15:13:36.915 INFO:tasks.ceph.mon.b:Restarting daemon 2026-02-17T15:13:36.916 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2026-02-17T15:13:36.917 INFO:tasks.ceph.mon.b:Started 2026-02-17T15:13:36.918 INFO:tasks.ceph.mon.c:Restarting daemon 2026-02-17T15:13:36.918 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2026-02-17T15:13:36.919 INFO:tasks.ceph.mon.c:Started 2026-02-17T15:13:36.920 DEBUG:teuthology.orchestra.run.trial024:> sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force 2026-02-17T15:13:47.452 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2026-02-17T15:13:47.452 INFO:tasks.ceph.mgr.x:Restarting daemon 2026-02-17T15:13:47.452 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2026-02-17T15:13:47.454 INFO:tasks.ceph.mgr.x:Started 2026-02-17T15:13:47.455 INFO:tasks.ceph:Setting crush tunables to default 2026-02-17T15:13:47.455 DEBUG:teuthology.orchestra.run.trial024:> sudo ceph --cluster ceph osd crush tunables default 2026-02-17T15:13:47.559 INFO:teuthology.orchestra.run.trial024.stderr:adjusted tunables profile to default 2026-02-17T15:13:47.561 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:47.555+0000 7f5527fa7280 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2026-02-17T15:13:47.569 INFO:tasks.ceph:check_enable_crimson: False 2026-02-17T15:13:47.569 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2026-02-17T15:13:47.569 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:47.569 DEBUG:teuthology.orchestra.run.trial024:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2026-02-17T15:13:47.579 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:47.579 DEBUG:teuthology.orchestra.run.trial024:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2026-02-17T15:13:47.588 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:47.583+0000 7f5527fa7280 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2026-02-17T15:13:47.596 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:47.596 DEBUG:teuthology.orchestra.run.trial024:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2026-02-17T15:13:47.625 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:47.619+0000 7f5527fa7280 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2026-02-17T15:13:47.633 DEBUG:teuthology.orchestra.run.trial024:> set -ex 2026-02-17T15:13:47.633 DEBUG:teuthology.orchestra.run.trial024:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2026-02-17T15:13:47.681 DEBUG:teuthology.orchestra.run.trial024:> sudo ceph --cluster ceph osd new ee39a229-ca94-4bb1-ab31-dd4d57c12e53 0 2026-02-17T15:13:47.714 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:47.711+0000 7f5527fa7280 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2026-02-17T15:13:47.795 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:47.791+0000 7f5527fa7280 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2026-02-17T15:13:47.811 INFO:teuthology.orchestra.run.trial024.stdout:0 2026-02-17T15:13:47.821 DEBUG:teuthology.orchestra.run.trial024:> sudo ceph --cluster ceph osd new 85f3808e-7c1e-4d80-98f5-1929c6a71b74 1 2026-02-17T15:13:47.833 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:47.827+0000 7f5527fa7280 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2026-02-17T15:13:47.909 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:47.903+0000 7f5527fa7280 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2026-02-17T15:13:47.918 INFO:teuthology.orchestra.run.trial024.stdout:1 2026-02-17T15:13:47.927 DEBUG:teuthology.orchestra.run.trial024:> sudo ceph --cluster ceph osd new 3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2 2 2026-02-17T15:13:48.067 INFO:teuthology.orchestra.run.trial024.stdout:2 2026-02-17T15:13:48.076 DEBUG:teuthology.orchestra.run.trial024:> sudo ceph --cluster ceph osd new ee3abe87-b5af-4c38-ae75-54a3f1e76cbf 3 2026-02-17T15:13:48.087 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:48.083+0000 7f5527fa7280 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2026-02-17T15:13:48.174 INFO:teuthology.orchestra.run.trial024.stdout:3 2026-02-17T15:13:48.183 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-17T15:13:48.183 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-17T15:13:48.231 INFO:tasks.ceph.osd.0:Started 2026-02-17T15:13:48.232 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-17T15:13:48.232 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-17T15:13:48.234 INFO:tasks.ceph.osd.1:Started 2026-02-17T15:13:48.234 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-17T15:13:48.234 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-17T15:13:48.236 INFO:tasks.ceph.osd.2:Started 2026-02-17T15:13:48.236 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-17T15:13:48.237 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2026-02-17T15:13:48.238 INFO:tasks.ceph.osd.3:Started 2026-02-17T15:13:48.239 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:13:48.334 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:13:48.334 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":6,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:13:48.171454+0000","last_up_change":"0.000000","last_in_change":"2026-02-17T15:13:48.171454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:13:48.345 INFO:tasks.ceph.ceph_manager.ceph:[] 2026-02-17T15:13:48.346 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-17T15:13:48.392 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:48.387+0000 7f5527fa7280 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2026-02-17T15:13:48.484 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:48.479+0000 7f5527fa7280 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2026-02-17T15:13:48.511 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:48.507+0000 7f5527fa7280 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2026-02-17T15:13:48.542 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:48.535+0000 7f5527fa7280 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2026-02-17T15:13:48.583 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:48.579+0000 7f5527fa7280 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2026-02-17T15:13:48.682 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:48.675+0000 7f5527fa7280 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2026-02-17T15:13:48.748 DEBUG:teuthology.orchestra.run.trial024:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-17T15:13:48.835 INFO:teuthology.misc.health.trial024.stdout: 2026-02-17T15:13:48.835 INFO:teuthology.misc.health.trial024.stdout:{"epoch":6,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:13:48.171454+0000","last_up_change":"0.000000","last_in_change":"2026-02-17T15:13:48.171454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:13:48.843 DEBUG:teuthology.misc:0 of 4 OSDs are up 2026-02-17T15:13:48.873 INFO:tasks.ceph.mgr.x.trial024.stderr:ERROR:root:Module 'xmltodict' is not installed. 2026-02-17T15:13:49.045 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:49.039+0000 7f5527fa7280 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2026-02-17T15:13:49.076 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:49.071+0000 7f5527fa7280 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2026-02-17T15:13:49.168 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:49.163+0000 7f5527fa7280 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2026-02-17T15:13:49.192 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:49.187+0000 7f5527fa7280 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2026-02-17T15:13:49.218 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:49.215+0000 7f5527fa7280 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2026-02-17T15:13:49.245 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:49.239+0000 7f5527fa7280 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2026-02-17T15:13:49.368 INFO:tasks.ceph.mgr.x.trial024.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-17T15:13:49.369 INFO:tasks.ceph.mgr.x.trial024.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-17T15:13:49.369 INFO:tasks.ceph.mgr.x.trial024.stderr: from numpy import show_config as show_numpy_config 2026-02-17T15:13:49.375 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:49.371+0000 7f5527fa7280 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2026-02-17T15:13:49.401 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:49.395+0000 7f5527fa7280 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2026-02-17T15:13:49.439 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:49.435+0000 7f5527fa7280 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2026-02-17T15:13:49.603 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:13:49.599+0000 7f00fc6768c0 -1 Falling back to public interface 2026-02-17T15:13:49.811 INFO:tasks.ceph.osd.1.trial024.stderr:2026-02-17T15:13:49.807+0000 7ff4e2dd08c0 -1 Falling back to public interface 2026-02-17T15:13:50.299 INFO:tasks.ceph.osd.0.trial024.stderr:2026-02-17T15:13:50.295+0000 7f6b314fd8c0 -1 Falling back to public interface 2026-02-17T15:13:50.371 INFO:tasks.ceph.osd.2.trial024.stderr:2026-02-17T15:13:50.367+0000 7f7b585bd8c0 -1 Falling back to public interface 2026-02-17T15:13:50.467 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:50.463+0000 7f54469f9640 -1 mgr.server handle_report got status from non-daemon mon.c 2026-02-17T15:13:50.467 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:13:50.463+0000 7f54469f9640 -1 mgr.server handle_report got status from non-daemon mon.b 2026-02-17T15:13:53.565 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:13:53.558+0000 7f00fc6768c0 -1 osd.3 0 log_to_monitors true 2026-02-17T15:13:54.151 INFO:tasks.ceph.osd.0.trial024.stderr:2026-02-17T15:13:54.146+0000 7f6b314fd8c0 -1 osd.0 0 log_to_monitors true 2026-02-17T15:13:54.272 INFO:tasks.ceph.osd.2.trial024.stderr:2026-02-17T15:13:54.266+0000 7f7b585bd8c0 -1 osd.2 0 log_to_monitors true 2026-02-17T15:13:54.535 INFO:tasks.ceph.osd.1.trial024.stderr:2026-02-17T15:13:54.530+0000 7ff4e2dd08c0 -1 osd.1 0 log_to_monitors true 2026-02-17T15:13:55.246 DEBUG:teuthology.orchestra.run.trial024:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-17T15:13:55.386 INFO:teuthology.misc.health.trial024.stdout: 2026-02-17T15:13:55.386 INFO:teuthology.misc.health.trial024.stdout:{"epoch":7,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:13:54.465589+0000","last_up_change":"0.000000","last_in_change":"2026-02-17T15:13:48.171454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:13:55.395 DEBUG:teuthology.misc:0 of 4 OSDs are up 2026-02-17T15:13:55.475 INFO:tasks.ceph.osd.2.trial024.stderr:2026-02-17T15:13:55.470+0000 7f7b54566640 -1 osd.2 0 waiting for initial osdmap 2026-02-17T15:13:55.476 INFO:tasks.ceph.osd.0.trial024.stderr:2026-02-17T15:13:55.470+0000 7f6b2d4a6640 -1 osd.0 0 waiting for initial osdmap 2026-02-17T15:13:55.479 INFO:tasks.ceph.osd.0.trial024.stderr:2026-02-17T15:13:55.474+0000 7f6b292e1640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T15:13:55.479 INFO:tasks.ceph.osd.2.trial024.stderr:2026-02-17T15:13:55.474+0000 7f7b503a1640 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T15:13:55.872 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:13:55.866+0000 7f00f861f640 -1 osd.3 0 waiting for initial osdmap 2026-02-17T15:13:55.876 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:13:55.870+0000 7f00f0452640 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T15:13:56.478 INFO:tasks.ceph.osd.1.trial024.stderr:2026-02-17T15:13:56.470+0000 7ff4ded79640 -1 osd.1 0 waiting for initial osdmap 2026-02-17T15:13:56.481 INFO:tasks.ceph.osd.1.trial024.stderr:2026-02-17T15:13:56.474+0000 7ff4dabb4640 -1 osd.1 9 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T15:14:01.798 DEBUG:teuthology.orchestra.run.trial024:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2026-02-17T15:14:01.939 INFO:teuthology.misc.health.trial024.stdout: 2026-02-17T15:14:01.939 INFO:teuthology.misc.health.trial024.stdout:{"epoch":13,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:14:00.483417+0000","last_up_change":"2026-02-17T15:13:57.474676+0000","last_in_change":"2026-02-17T15:13:48.171454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":4268245710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":4268245710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":4268245710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":4268245710}]},"public_addr":"10.20.193.24:6809/4268245710","cluster_addr":"10.20.193.24:6810/4268245710","heartbeat_back_addr":"10.20.193.24:6812/4268245710","heartbeat_front_addr":"10.20.193.24:6811/4268245710","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":3281207742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":3281207742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":3281207742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":3281207742}]},"public_addr":"10.20.193.24:6805/3281207742","cluster_addr":"10.20.193.24:6806/3281207742","heartbeat_back_addr":"10.20.193.24:6808/3281207742","heartbeat_front_addr":"10.20.193.24:6807/3281207742","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":1012255776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":1012255776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":1012255776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":1012255776}]},"public_addr":"10.20.193.24:6813/1012255776","cluster_addr":"10.20.193.24:6814/1012255776","heartbeat_back_addr":"10.20.193.24:6816/1012255776","heartbeat_front_addr":"10.20.193.24:6815/1012255776","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":3626123643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":3626123643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":3626123643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":3626123643}]},"public_addr":"10.20.193.24:6801/3626123643","cluster_addr":"10.20.193.24:6802/3626123643","heartbeat_back_addr":"10.20.193.24:6804/3626123643","heartbeat_front_addr":"10.20.193.24:6803/3626123643","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:14:01.949 DEBUG:teuthology.misc:4 of 4 OSDs are up 2026-02-17T15:14:01.949 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2026-02-17T15:14:01.949 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2026-02-17T15:14:01.949 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2026-02-17T15:14:01.950 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2026-02-17T15:14:01.950 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-17T15:14:02.113 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:02.122 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":4,"flags":0,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6800","nonce":388367308}]},"active_addr":"10.20.193.24:6800/388367308","active_change":"2026-02-17T15:13:49.457900+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number. if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.24:0","nonce":1686801553}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.24:0","nonce":1023083890}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.24:0","nonce":1687970201}]}]} 2026-02-17T15:14:02.126 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2026-02-17T15:14:02.126 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2026-02-17T15:14:02.126 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:14:02.268 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:02.268 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":13,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:14:00.483417+0000","last_up_change":"2026-02-17T15:13:57.474676+0000","last_in_change":"2026-02-17T15:13:48.171454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":4268245710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":4268245710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":4268245710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":4268245710}]},"public_addr":"10.20.193.24:6809/4268245710","cluster_addr":"10.20.193.24:6810/4268245710","heartbeat_back_addr":"10.20.193.24:6812/4268245710","heartbeat_front_addr":"10.20.193.24:6811/4268245710","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":3281207742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":3281207742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":3281207742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":3281207742}]},"public_addr":"10.20.193.24:6805/3281207742","cluster_addr":"10.20.193.24:6806/3281207742","heartbeat_back_addr":"10.20.193.24:6808/3281207742","heartbeat_front_addr":"10.20.193.24:6807/3281207742","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":1012255776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":1012255776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":1012255776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":1012255776}]},"public_addr":"10.20.193.24:6813/1012255776","cluster_addr":"10.20.193.24:6814/1012255776","heartbeat_back_addr":"10.20.193.24:6816/1012255776","heartbeat_front_addr":"10.20.193.24:6815/1012255776","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":3626123643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":3626123643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":3626123643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":3626123643}]},"public_addr":"10.20.193.24:6801/3626123643","cluster_addr":"10.20.193.24:6802/3626123643","heartbeat_back_addr":"10.20.193.24:6804/3626123643","heartbeat_front_addr":"10.20.193.24:6803/3626123643","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:14:02.278 INFO:tasks.ceph.ceph_manager.ceph:all up! 2026-02-17T15:14:02.278 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:14:02.420 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:02.420 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":13,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:14:00.483417+0000","last_up_change":"2026-02-17T15:13:57.474676+0000","last_in_change":"2026-02-17T15:13:48.171454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":4268245710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":4268245710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":4268245710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":4268245710}]},"public_addr":"10.20.193.24:6809/4268245710","cluster_addr":"10.20.193.24:6810/4268245710","heartbeat_back_addr":"10.20.193.24:6812/4268245710","heartbeat_front_addr":"10.20.193.24:6811/4268245710","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":3281207742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":3281207742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":3281207742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":3281207742}]},"public_addr":"10.20.193.24:6805/3281207742","cluster_addr":"10.20.193.24:6806/3281207742","heartbeat_back_addr":"10.20.193.24:6808/3281207742","heartbeat_front_addr":"10.20.193.24:6807/3281207742","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":1012255776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":1012255776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":1012255776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":1012255776}]},"public_addr":"10.20.193.24:6813/1012255776","cluster_addr":"10.20.193.24:6814/1012255776","heartbeat_back_addr":"10.20.193.24:6816/1012255776","heartbeat_front_addr":"10.20.193.24:6815/1012255776","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":3626123643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":3626123643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":3626123643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":3626123643}]},"public_addr":"10.20.193.24:6801/3626123643","cluster_addr":"10.20.193.24:6802/3626123643","heartbeat_back_addr":"10.20.193.24:6804/3626123643","heartbeat_front_addr":"10.20.193.24:6803/3626123643","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:14:02.431 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:14:02.431 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:14:02.432 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:14:02.432 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:14:02.517 INFO:teuthology.orchestra.run.trial024.stdout:42949672962 2026-02-17T15:14:02.518 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:14:02.519 INFO:teuthology.orchestra.run.trial024.stdout:38654705667 2026-02-17T15:14:02.520 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:14:02.521 INFO:teuthology.orchestra.run.trial024.stdout:38654705667 2026-02-17T15:14:02.521 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:14:02.522 INFO:teuthology.orchestra.run.trial024.stdout:38654705667 2026-02-17T15:14:02.522 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:14:02.709 INFO:teuthology.orchestra.run.trial024.stdout:38654705665 2026-02-17T15:14:02.712 INFO:teuthology.orchestra.run.trial024.stdout:42949672961 2026-02-17T15:14:02.713 INFO:teuthology.orchestra.run.trial024.stdout:38654705665 2026-02-17T15:14:02.719 INFO:teuthology.orchestra.run.trial024.stdout:38654705665 2026-02-17T15:14:02.721 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705665 for osd.2 2026-02-17T15:14:02.724 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672962 got 42949672961 for osd.1 2026-02-17T15:14:02.725 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705665 for osd.3 2026-02-17T15:14:02.732 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705665 for osd.0 2026-02-17T15:14:03.721 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:14:03.725 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:14:03.725 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:14:03.732 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:14:03.869 INFO:teuthology.orchestra.run.trial024.stdout:38654705667 2026-02-17T15:14:03.882 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.2 2026-02-17T15:14:03.882 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:03.911 INFO:teuthology.orchestra.run.trial024.stdout:38654705667 2026-02-17T15:14:03.915 INFO:teuthology.orchestra.run.trial024.stdout:42949672962 2026-02-17T15:14:03.919 INFO:teuthology.orchestra.run.trial024.stdout:38654705667 2026-02-17T15:14:03.923 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.0 2026-02-17T15:14:03.923 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:03.927 INFO:tasks.ceph.ceph_manager.ceph:need seq 42949672962 got 42949672962 for osd.1 2026-02-17T15:14:03.927 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:03.931 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705667 got 38654705667 for osd.3 2026-02-17T15:14:03.932 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:03.932 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-17T15:14:03.932 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:14:04.120 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:04.121 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:14:04.130 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2026-02-17T15:14:03.462853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":2,"kb":377487360,"kb_used":112516,"kb_used_data":488,"kb_used_omap":6,"kb_used_meta":111929,"kb_avail":377374844,"statfs":{"total":386547056640,"available":386431840256,"internally_reserved":0,"allocated":499712,"data_stored":765405,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6374,"internal_metadata":114616090},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"12'17","reported_seq":35,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-17T15:14:00.836454+0000","last_change":"2026-02-17T15:13:59.954581+0000","last_active":"2026-02-17T15:14:00.836454+0000","last_peered":"2026-02-17T15:14:00.836454+0000","last_clean":"2026-02-17T15:14:00.836454+0000","last_became_active":"2026-02-17T15:13:59.953788+0000","last_became_peered":"2026-02-17T15:13:59.953788+0000","last_unstale":"2026-02-17T15:14:00.836454+0000","last_undegraded":"2026-02-17T15:14:00.836454+0000","last_fullsized":"2026-02-17T15:14:00.836454+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T00:12:00.574225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27100,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":139264,"data_stored":356204,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31240,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340600,"statfs":{"total":96636764160,"available":96604774400,"internally_reserved":0,"allocated":114688,"data_stored":28012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344764,"statfs":{"total":96636764160,"available":96609038336,"internally_reserved":0,"allocated":114688,"data_stored":28012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27100,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":131072,"data_stored":353177,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T15:14:04.131 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:14:04.281 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:04.282 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:14:04.292 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2026-02-17T15:14:03.462853+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":1,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":2,"kb":377487360,"kb_used":112516,"kb_used_data":488,"kb_used_omap":6,"kb_used_meta":111929,"kb_avail":377374844,"statfs":{"total":386547056640,"available":386431840256,"internally_reserved":0,"allocated":499712,"data_stored":765405,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6374,"internal_metadata":114616090},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.0","version":"12'17","reported_seq":35,"reported_epoch":13,"state":"active+clean","last_fresh":"2026-02-17T15:14:00.836454+0000","last_change":"2026-02-17T15:13:59.954581+0000","last_active":"2026-02-17T15:14:00.836454+0000","last_peered":"2026-02-17T15:14:00.836454+0000","last_clean":"2026-02-17T15:14:00.836454+0000","last_became_active":"2026-02-17T15:13:59.953788+0000","last_became_peered":"2026-02-17T15:13:59.953788+0000","last_unstale":"2026-02-17T15:14:00.836454+0000","last_undegraded":"2026-02-17T15:14:00.836454+0000","last_fullsized":"2026-02-17T15:14:00.836454+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T00:12:00.574225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672962,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27100,"kb_used_data":136,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":139264,"data_stored":356204,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31240,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340600,"statfs":{"total":96636764160,"available":96604774400,"internally_reserved":0,"allocated":114688,"data_stored":28012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27076,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344764,"statfs":{"total":96636764160,"available":96609038336,"internally_reserved":0,"allocated":114688,"data_stored":28012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705667,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27100,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344740,"statfs":{"total":96636764160,"available":96609013760,"internally_reserved":0,"allocated":131072,"data_stored":353177,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T15:14:04.295 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-17T15:14:04.295 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2026-02-17T15:14:04.295 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2026-02-17T15:14:04.295 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2026-02-17T15:14:04.460 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:04.460 INFO:teuthology.orchestra.run.trial024.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-17T15:14:04.470 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2026-02-17T15:14:04.471 INFO:teuthology.run_tasks:Running task ec_inconsistent_hinfo... 2026-02-17T15:14:04.493 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:14:04.639 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:04.639 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":13,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:14:00.483417+0000","last_up_change":"2026-02-17T15:13:57.474676+0000","last_in_change":"2026-02-17T15:13:48.171454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":4268245710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":4268245710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":4268245710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":4268245710}]},"public_addr":"10.20.193.24:6809/4268245710","cluster_addr":"10.20.193.24:6810/4268245710","heartbeat_back_addr":"10.20.193.24:6812/4268245710","heartbeat_front_addr":"10.20.193.24:6811/4268245710","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":3281207742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":3281207742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":3281207742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":3281207742}]},"public_addr":"10.20.193.24:6805/3281207742","cluster_addr":"10.20.193.24:6806/3281207742","heartbeat_back_addr":"10.20.193.24:6808/3281207742","heartbeat_front_addr":"10.20.193.24:6807/3281207742","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":1012255776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":1012255776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":1012255776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":1012255776}]},"public_addr":"10.20.193.24:6813/1012255776","cluster_addr":"10.20.193.24:6814/1012255776","heartbeat_back_addr":"10.20.193.24:6816/1012255776","heartbeat_front_addr":"10.20.193.24:6815/1012255776","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":3626123643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":3626123643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":3626123643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":3626123643}]},"public_addr":"10.20.193.24:6801/3626123643","cluster_addr":"10.20.193.24:6802/3626123643","heartbeat_back_addr":"10.20.193.24:6804/3626123643","heartbeat_front_addr":"10.20.193.24:6803/3626123643","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:14:04.651 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2026-02-17T15:13:57.498851+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4, 'score_stable': 4, 'optimal_score': 0.5, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2026-02-17T15:14:04.651 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2026-02-17T15:14:04.798 INFO:teuthology.orchestra.run.trial024.stdout:pg_num: 1 2026-02-17T15:14:04.810 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set backfill_unfound k=2 m=1 crush-failure-domain=osd 2026-02-17T15:14:05.512 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:creating pool_name unique_pool_0 2026-02-17T15:14:05.513 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 1 1 erasure backfill_unfound 2026-02-17T15:14:07.508 INFO:teuthology.orchestra.run.trial024.stderr:pool 'unique_pool_0' created 2026-02-17T15:14:07.521 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 min_size 2 2026-02-17T15:14:08.513 INFO:teuthology.orchestra.run.trial024.stderr:set pool 2 min_size to 2 2026-02-17T15:14:08.527 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2026-02-17T15:14:09.515 INFO:teuthology.orchestra.run.trial024.stderr:enabled application 'rados' on pool 'unique_pool_0' 2026-02-17T15:14:10.528 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set unique_pool_0 pg_autoscale_mode off 2026-02-17T15:14:11.519 INFO:teuthology.orchestra.run.trial024.stderr:set pool 2 pg_autoscale_mode to off 2026-02-17T15:14:11.531 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:14:11.532 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:14:11.532 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:14:11.533 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:14:11.617 INFO:teuthology.orchestra.run.trial024.stdout:38654705670 2026-02-17T15:14:11.618 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:14:11.619 INFO:teuthology.orchestra.run.trial024.stdout:42949672965 2026-02-17T15:14:11.620 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:14:11.621 INFO:teuthology.orchestra.run.trial024.stdout:38654705670 2026-02-17T15:14:11.622 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:14:11.624 INFO:teuthology.orchestra.run.trial024.stdout:38654705670 2026-02-17T15:14:11.625 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:14:11.768 INFO:teuthology.orchestra.run.trial024.stdout:42949672964 2026-02-17T15:14:11.769 INFO:teuthology.orchestra.run.trial024.stdout:38654705668 2026-02-17T15:14:11.781 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672965 got 42949672964 for osd.1 2026-02-17T15:14:11.782 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705670 got 38654705668 for osd.3 2026-02-17T15:14:11.819 INFO:teuthology.orchestra.run.trial024.stdout:38654705668 2026-02-17T15:14:11.824 INFO:teuthology.orchestra.run.trial024.stdout:38654705668 2026-02-17T15:14:11.831 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705670 got 38654705668 for osd.2 2026-02-17T15:14:11.835 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705670 got 38654705668 for osd.0 2026-02-17T15:14:12.781 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:14:12.783 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:14:12.832 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:14:12.836 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:14:12.925 INFO:teuthology.orchestra.run.trial024.stdout:42949672964 2026-02-17T15:14:12.929 INFO:teuthology.orchestra.run.trial024.stdout:38654705668 2026-02-17T15:14:12.937 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672965 got 42949672964 for osd.1 2026-02-17T15:14:12.940 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705670 got 38654705668 for osd.3 2026-02-17T15:14:12.996 INFO:teuthology.orchestra.run.trial024.stdout:38654705668 2026-02-17T15:14:13.008 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705670 got 38654705668 for osd.2 2026-02-17T15:14:13.022 INFO:teuthology.orchestra.run.trial024.stdout:38654705668 2026-02-17T15:14:13.034 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705670 got 38654705668 for osd.0 2026-02-17T15:14:13.938 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:14:13.941 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:14:14.009 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:14:14.034 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:14:14.096 INFO:teuthology.orchestra.run.trial024.stdout:42949672965 2026-02-17T15:14:14.111 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672965 got 42949672965 for osd.1 2026-02-17T15:14:14.112 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:14.133 INFO:teuthology.orchestra.run.trial024.stdout:38654705670 2026-02-17T15:14:14.145 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705670 got 38654705670 for osd.3 2026-02-17T15:14:14.145 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:14.172 INFO:teuthology.orchestra.run.trial024.stdout:38654705670 2026-02-17T15:14:14.184 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705670 got 38654705670 for osd.2 2026-02-17T15:14:14.184 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:14.219 INFO:teuthology.orchestra.run.trial024.stdout:38654705670 2026-02-17T15:14:14.231 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705670 got 38654705670 for osd.0 2026-02-17T15:14:14.231 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:14.231 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for clean 2026-02-17T15:14:14.231 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:14:14.416 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:14.416 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:14:14.426 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2026-02-17T15:14:13.464312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":112680,"kb_used_data":656,"kb_used_omap":6,"kb_used_meta":111929,"kb_avail":377374680,"statfs":{"total":386547056640,"available":386431672320,"internally_reserved":0,"allocated":671744,"data_stored":842224,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6374,"internal_metadata":114616090},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.958138"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-17T15:14:11.518193+0000","last_change":"2026-02-17T15:14:08.514335+0000","last_active":"2026-02-17T15:14:11.518193+0000","last_peered":"2026-02-17T15:14:11.518193+0000","last_clean":"2026-02-17T15:14:11.518193+0000","last_became_active":"2026-02-17T15:14:08.514135+0000","last_became_peered":"2026-02-17T15:14:08.514135+0000","last_unstale":"2026-02-17T15:14:11.518193+0000","last_undegraded":"2026-02-17T15:14:11.518193+0000","last_fullsized":"2026-02-17T15:14:11.518193+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:14:07.503000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:14:07.503000+0000","last_clean_scrub_stamp":"2026-02-17T15:14:07.503000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T21:46:33.332871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":45,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-17T15:14:09.766615+0000","last_change":"2026-02-17T15:13:59.954581+0000","last_active":"2026-02-17T15:14:09.766615+0000","last_peered":"2026-02-17T15:14:09.766615+0000","last_clean":"2026-02-17T15:14:09.766615+0000","last_became_active":"2026-02-17T15:13:59.953788+0000","last_became_peered":"2026-02-17T15:13:59.953788+0000","last_unstale":"2026-02-17T15:14:09.766615+0000","last_undegraded":"2026-02-17T15:14:09.766615+0000","last_fullsized":"2026-02-17T15:14:09.766615+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T00:12:00.574225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":180224,"data_stored":374652,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31280,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340560,"statfs":{"total":96636764160,"available":96604733440,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27116,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":180224,"data_stored":374652,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T15:14:14.426 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:14:14.568 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:14.568 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:14:14.581 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2026-02-17T15:14:13.464312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":112680,"kb_used_data":656,"kb_used_omap":6,"kb_used_meta":111929,"kb_avail":377374680,"statfs":{"total":386547056640,"available":386431672320,"internally_reserved":0,"allocated":671744,"data_stored":842224,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6374,"internal_metadata":114616090},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.958138"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-17T15:14:11.518193+0000","last_change":"2026-02-17T15:14:08.514335+0000","last_active":"2026-02-17T15:14:11.518193+0000","last_peered":"2026-02-17T15:14:11.518193+0000","last_clean":"2026-02-17T15:14:11.518193+0000","last_became_active":"2026-02-17T15:14:08.514135+0000","last_became_peered":"2026-02-17T15:14:08.514135+0000","last_unstale":"2026-02-17T15:14:11.518193+0000","last_undegraded":"2026-02-17T15:14:11.518193+0000","last_fullsized":"2026-02-17T15:14:11.518193+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:14:07.503000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:14:07.503000+0000","last_clean_scrub_stamp":"2026-02-17T15:14:07.503000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T21:46:33.332871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":45,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-17T15:14:09.766615+0000","last_change":"2026-02-17T15:13:59.954581+0000","last_active":"2026-02-17T15:14:09.766615+0000","last_peered":"2026-02-17T15:14:09.766615+0000","last_clean":"2026-02-17T15:14:09.766615+0000","last_became_active":"2026-02-17T15:13:59.953788+0000","last_became_peered":"2026-02-17T15:13:59.953788+0000","last_unstale":"2026-02-17T15:14:09.766615+0000","last_undegraded":"2026-02-17T15:14:09.766615+0000","last_fullsized":"2026-02-17T15:14:09.766615+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T00:12:00.574225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":180224,"data_stored":374652,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31280,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340560,"statfs":{"total":96636764160,"available":96604733440,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27116,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":180224,"data_stored":374652,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T15:14:14.582 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:clean! 2026-02-17T15:14:14.582 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:14:14.728 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:14.728 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":19,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:14:11.514089+0000","last_up_change":"2026-02-17T15:13:57.474676+0000","last_in_change":"2026-02-17T15:13:48.171454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-17T15:14:06.508581+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":4268245710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":4268245710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":4268245710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":4268245710}]},"public_addr":"10.20.193.24:6809/4268245710","cluster_addr":"10.20.193.24:6810/4268245710","heartbeat_back_addr":"10.20.193.24:6812/4268245710","heartbeat_front_addr":"10.20.193.24:6811/4268245710","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":3281207742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":3281207742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":3281207742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":3281207742}]},"public_addr":"10.20.193.24:6805/3281207742","cluster_addr":"10.20.193.24:6806/3281207742","heartbeat_back_addr":"10.20.193.24:6808/3281207742","heartbeat_front_addr":"10.20.193.24:6807/3281207742","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":1012255776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":1012255776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":1012255776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":1012255776}]},"public_addr":"10.20.193.24:6813/1012255776","cluster_addr":"10.20.193.24:6814/1012255776","heartbeat_back_addr":"10.20.193.24:6816/1012255776","heartbeat_front_addr":"10.20.193.24:6815/1012255776","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":3626123643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":3626123643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":3626123643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":3626123643}]},"public_addr":"10.20.193.24:6801/3626123643","cluster_addr":"10.20.193.24:6802/3626123643","heartbeat_back_addr":"10.20.193.24:6804/3626123643","heartbeat_front_addr":"10.20.193.24:6803/3626123643","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:14:14.738 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:14:14.883 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:14.883 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:14:14.895 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2026-02-17T15:14:13.464312+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":112680,"kb_used_data":656,"kb_used_omap":6,"kb_used_meta":111929,"kb_avail":377374680,"statfs":{"total":386547056640,"available":386431672320,"internally_reserved":0,"allocated":671744,"data_stored":842224,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6374,"internal_metadata":114616090},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.958138"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-17T15:14:11.518193+0000","last_change":"2026-02-17T15:14:08.514335+0000","last_active":"2026-02-17T15:14:11.518193+0000","last_peered":"2026-02-17T15:14:11.518193+0000","last_clean":"2026-02-17T15:14:11.518193+0000","last_became_active":"2026-02-17T15:14:08.514135+0000","last_became_peered":"2026-02-17T15:14:08.514135+0000","last_unstale":"2026-02-17T15:14:11.518193+0000","last_undegraded":"2026-02-17T15:14:11.518193+0000","last_fullsized":"2026-02-17T15:14:11.518193+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:14:07.503000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:14:07.503000+0000","last_clean_scrub_stamp":"2026-02-17T15:14:07.503000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T21:46:33.332871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":45,"reported_epoch":18,"state":"active+clean","last_fresh":"2026-02-17T15:14:09.766615+0000","last_change":"2026-02-17T15:13:59.954581+0000","last_active":"2026-02-17T15:14:09.766615+0000","last_peered":"2026-02-17T15:14:09.766615+0000","last_clean":"2026-02-17T15:14:09.766615+0000","last_became_active":"2026-02-17T15:13:59.953788+0000","last_became_peered":"2026-02-17T15:13:59.953788+0000","last_unstale":"2026-02-17T15:14:09.766615+0000","last_undegraded":"2026-02-17T15:14:09.766615+0000","last_fullsized":"2026-02-17T15:14:09.766615+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T00:12:00.574225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672965,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":180224,"data_stored":374652,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31280,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340560,"statfs":{"total":96636764160,"available":96604733440,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27116,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27144,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344696,"statfs":{"total":96636764160,"available":96608968704,"internally_reserved":0,"allocated":180224,"data_stored":374652,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T15:14:14.896 INFO:tasks.ec_inconsistent_hinfo:acting=[3, 1, 2] 2026-02-17T15:14:14.896 INFO:tasks.util.rados:rados -p unique_pool_0 put dummy /etc/group 2026-02-17T15:14:14.896 DEBUG:teuthology.orchestra.run.trial024:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put dummy /etc/group 2026-02-17T15:14:14.925 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:14:14.925 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:14:15.057 INFO:teuthology.orchestra.run.trial024.stdout:42949672967 2026-02-17T15:14:15.058 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:14:15.059 INFO:teuthology.orchestra.run.trial024.stdout:38654705671 2026-02-17T15:14:15.060 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:14:15.247 INFO:teuthology.orchestra.run.trial024.stdout:38654705670 2026-02-17T15:14:15.253 INFO:teuthology.orchestra.run.trial024.stdout:42949672965 2026-02-17T15:14:15.258 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705671 got 38654705670 for osd.0 2026-02-17T15:14:15.264 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672967 got 42949672965 for osd.1 2026-02-17T15:14:16.259 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:14:16.265 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:14:16.420 INFO:teuthology.orchestra.run.trial024.stdout:38654705671 2026-02-17T15:14:16.431 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705671 got 38654705671 for osd.0 2026-02-17T15:14:16.431 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:16.442 INFO:teuthology.orchestra.run.trial024.stdout:42949672967 2026-02-17T15:14:16.452 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 42949672967 got 42949672967 for osd.1 2026-02-17T15:14:16.452 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:16.452 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for recovery to complete 2026-02-17T15:14:16.452 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:14:16.634 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:16.635 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:14:16.645 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2026-02-17T15:14:15.464599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":112704,"kb_used_data":672,"kb_used_omap":6,"kb_used_meta":111929,"kb_avail":377374656,"statfs":{"total":386547056640,"available":386431647744,"internally_reserved":0,"allocated":688128,"data_stored":849798,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6374,"internal_metadata":114616090},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.953838"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-17T15:14:11.518193+0000","last_change":"2026-02-17T15:14:08.514335+0000","last_active":"2026-02-17T15:14:11.518193+0000","last_peered":"2026-02-17T15:14:11.518193+0000","last_clean":"2026-02-17T15:14:11.518193+0000","last_became_active":"2026-02-17T15:14:08.514135+0000","last_became_peered":"2026-02-17T15:14:08.514135+0000","last_unstale":"2026-02-17T15:14:11.518193+0000","last_undegraded":"2026-02-17T15:14:11.518193+0000","last_fullsized":"2026-02-17T15:14:11.518193+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:14:07.503000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:14:07.503000+0000","last_clean_scrub_stamp":"2026-02-17T15:14:07.503000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T21:46:33.332871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":47,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-17T15:14:12.274517+0000","last_change":"2026-02-17T15:13:59.954581+0000","last_active":"2026-02-17T15:14:12.274517+0000","last_peered":"2026-02-17T15:14:12.274517+0000","last_clean":"2026-02-17T15:14:12.274517+0000","last_became_active":"2026-02-17T15:13:59.953788+0000","last_became_peered":"2026-02-17T15:13:59.953788+0000","last_unstale":"2026-02-17T15:14:12.274517+0000","last_undegraded":"2026-02-17T15:14:12.274517+0000","last_fullsized":"2026-02-17T15:14:12.274517+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T00:12:00.574225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8192,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672967,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":188416,"data_stored":378439,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31280,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340560,"statfs":{"total":96636764160,"available":96604733440,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27116,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":188416,"data_stored":378439,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8192,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T15:14:16.646 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:14:16.787 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:16.787 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:14:16.797 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2026-02-17T15:14:15.464599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":112704,"kb_used_data":672,"kb_used_omap":6,"kb_used_meta":111929,"kb_avail":377374656,"statfs":{"total":386547056640,"available":386431647744,"internally_reserved":0,"allocated":688128,"data_stored":849798,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":6374,"internal_metadata":114616090},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.953838"},"pg_stats":[{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-17T15:14:11.518193+0000","last_change":"2026-02-17T15:14:08.514335+0000","last_active":"2026-02-17T15:14:11.518193+0000","last_peered":"2026-02-17T15:14:11.518193+0000","last_clean":"2026-02-17T15:14:11.518193+0000","last_became_active":"2026-02-17T15:14:08.514135+0000","last_became_peered":"2026-02-17T15:14:08.514135+0000","last_unstale":"2026-02-17T15:14:11.518193+0000","last_undegraded":"2026-02-17T15:14:11.518193+0000","last_fullsized":"2026-02-17T15:14:11.518193+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:14:07.503000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:14:07.503000+0000","last_clean_scrub_stamp":"2026-02-17T15:14:07.503000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T21:46:33.332871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":47,"reported_epoch":19,"state":"active+clean","last_fresh":"2026-02-17T15:14:12.274517+0000","last_change":"2026-02-17T15:13:59.954581+0000","last_active":"2026-02-17T15:14:12.274517+0000","last_peered":"2026-02-17T15:14:12.274517+0000","last_clean":"2026-02-17T15:14:12.274517+0000","last_became_active":"2026-02-17T15:13:59.953788+0000","last_became_peered":"2026-02-17T15:13:59.953788+0000","last_unstale":"2026-02-17T15:14:12.274517+0000","last_undegraded":"2026-02-17T15:14:12.274517+0000","last_fullsized":"2026-02-17T15:14:12.274517+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T00:12:00.574225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8192,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":10,"seq":42949672967,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":188416,"data_stored":378439,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31280,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340560,"statfs":{"total":96636764160,"available":96604733440,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27116,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344724,"statfs":{"total":96636764160,"available":96608997376,"internally_reserved":0,"allocated":155648,"data_stored":46460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705671,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":184,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":188416,"data_stored":378439,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":8192,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T15:14:16.798 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:recovered! 2026-02-17T15:14:16.798 DEBUG:tasks.ec_inconsistent_hinfo:create test object 2026-02-17T15:14:16.798 INFO:tasks.util.rados:rados -p unique_pool_0 put test /etc/group 2026-02-17T15:14:16.798 DEBUG:teuthology.orchestra.run.trial024:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 put test /etc/group 2026-02-17T15:14:16.825 INFO:tasks.ec_inconsistent_hinfo:remove test object hash info from osd.1 shard and test deep-scrub and repair 2026-02-17T15:14:16.826 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:14:17.009 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:17.009 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":19,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:14:11.514089+0000","last_up_change":"2026-02-17T15:13:57.474676+0000","last_in_change":"2026-02-17T15:13:48.171454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-17T15:14:06.508581+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":4268245710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":4268245710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":4268245710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":4268245710}]},"public_addr":"10.20.193.24:6809/4268245710","cluster_addr":"10.20.193.24:6810/4268245710","heartbeat_back_addr":"10.20.193.24:6812/4268245710","heartbeat_front_addr":"10.20.193.24:6811/4268245710","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":10,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":3281207742}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":3281207742}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":3281207742}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":3281207742}]},"public_addr":"10.20.193.24:6805/3281207742","cluster_addr":"10.20.193.24:6806/3281207742","heartbeat_back_addr":"10.20.193.24:6808/3281207742","heartbeat_front_addr":"10.20.193.24:6807/3281207742","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":1012255776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":1012255776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":1012255776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":1012255776}]},"public_addr":"10.20.193.24:6813/1012255776","cluster_addr":"10.20.193.24:6814/1012255776","heartbeat_back_addr":"10.20.193.24:6816/1012255776","heartbeat_front_addr":"10.20.193.24:6815/1012255776","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":3626123643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":3626123643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":3626123643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":3626123643}]},"public_addr":"10.20.193.24:6801/3626123643","cluster_addr":"10.20.193.24:6802/3626123643","heartbeat_back_addr":"10.20.193.24:6804/3626123643","heartbeat_front_addr":"10.20.193.24:6803/3626123643","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:14:17.020 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-02-17T15:14:17.161 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:17.171 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":19,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,1,2],"up_primary":3,"acting":[3,1,2],"acting_primary":3} 2026-02-17T15:14:17.172 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-17T15:14:17.172 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T15:14:17.254 INFO:tasks.ceph.osd.1:Stopped 2026-02-17T15:14:17.254 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 "$object" rm-attr hinfo_key 2026-02-17T15:14:17.254 DEBUG:teuthology.orchestra.run.trial024:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 --op list |grep '"'"'"oid":"test"'"'"') 2026-02-17T15:14:17.254 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 "$object" rm-attr hinfo_key' 2026-02-17T15:14:17.259 INFO:teuthology.orchestra.run.trial024.stderr:+ '[' -z '' ']' 2026-02-17T15:14:17.259 INFO:teuthology.orchestra.run.trial024.stderr:+ return 2026-02-17T15:14:17.259 INFO:teuthology.orchestra.run.trial024.stderr:+ case $- in 2026-02-17T15:14:17.259 INFO:teuthology.orchestra.run.trial024.stderr:+ return 2026-02-17T15:14:17.260 INFO:teuthology.orchestra.run.trial024.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 --op list 2026-02-17T15:14:17.260 INFO:teuthology.orchestra.run.trial024.stderr:++ grep '"oid":"test"' 2026-02-17T15:14:18.144 INFO:teuthology.orchestra.run.trial024.stderr:+ object='["2.0s1",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":1,"max":0}]' 2026-02-17T15:14:18.145 INFO:teuthology.orchestra.run.trial024.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --pgid 2.0s1 '["2.0s1",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":1,"max":0}]' rm-attr hinfo_key 2026-02-17T15:14:20.144 INFO:tasks.ceph.osd.1:Restarting daemon 2026-02-17T15:14:20.144 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2026-02-17T15:14:20.188 INFO:tasks.ceph.osd.1:Started 2026-02-17T15:14:20.188 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-17T15:14:20.247 INFO:teuthology.orchestra.run.trial024.stderr:no valid command found; 10 closest matches: 2026-02-17T15:14:20.248 INFO:teuthology.orchestra.run.trial024.stderr:0 2026-02-17T15:14:20.248 INFO:teuthology.orchestra.run.trial024.stderr:1 2026-02-17T15:14:20.248 INFO:teuthology.orchestra.run.trial024.stderr:2 2026-02-17T15:14:20.248 INFO:teuthology.orchestra.run.trial024.stderr:abort 2026-02-17T15:14:20.248 INFO:teuthology.orchestra.run.trial024.stderr:assert 2026-02-17T15:14:20.248 INFO:teuthology.orchestra.run.trial024.stderr:config diff 2026-02-17T15:14:20.248 INFO:teuthology.orchestra.run.trial024.stderr:config diff get 2026-02-17T15:14:20.248 INFO:teuthology.orchestra.run.trial024.stderr:config get 2026-02-17T15:14:20.248 INFO:teuthology.orchestra.run.trial024.stderr:config help [] 2026-02-17T15:14:20.248 INFO:teuthology.orchestra.run.trial024.stderr:config set ... 2026-02-17T15:14:20.249 INFO:teuthology.orchestra.run.trial024.stderr:admin_socket: invalid command 2026-02-17T15:14:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-17T15:14:20.250 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-17T15:14:21.436 INFO:tasks.ceph.osd.1.trial024.stderr:2026-02-17T15:14:21.430+0000 7fa5df0728c0 -1 Falling back to public interface 2026-02-17T15:14:25.252 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-17T15:14:25.266 INFO:tasks.ceph.osd.1.trial024.stderr:2026-02-17T15:14:25.257+0000 7fa5df0728c0 -1 osd.1 19 log_to_monitors true 2026-02-17T15:14:25.311 INFO:teuthology.orchestra.run.trial024.stderr:no valid command found; 10 closest matches: 2026-02-17T15:14:25.312 INFO:teuthology.orchestra.run.trial024.stderr:0 2026-02-17T15:14:25.312 INFO:teuthology.orchestra.run.trial024.stderr:1 2026-02-17T15:14:25.312 INFO:teuthology.orchestra.run.trial024.stderr:2 2026-02-17T15:14:25.312 INFO:teuthology.orchestra.run.trial024.stderr:abort 2026-02-17T15:14:25.312 INFO:teuthology.orchestra.run.trial024.stderr:assert 2026-02-17T15:14:25.312 INFO:teuthology.orchestra.run.trial024.stderr:bluefs debug_inject_read_zeros 2026-02-17T15:14:25.312 INFO:teuthology.orchestra.run.trial024.stderr:bluefs files list 2026-02-17T15:14:25.312 INFO:teuthology.orchestra.run.trial024.stderr:bluefs stats 2026-02-17T15:14:25.312 INFO:teuthology.orchestra.run.trial024.stderr:bluestore allocator dump block 2026-02-17T15:14:25.313 INFO:teuthology.orchestra.run.trial024.stderr:bluestore allocator fragmentation block 2026-02-17T15:14:25.313 INFO:teuthology.orchestra.run.trial024.stderr:admin_socket: invalid command 2026-02-17T15:14:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-17T15:14:25.314 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2026-02-17T15:14:25.326 INFO:tasks.ceph.osd.1.trial024.stderr:2026-02-17T15:14:25.317+0000 7fa5d6e56640 -1 osd.1 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T15:14:30.315 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2026-02-17T15:14:30.382 INFO:teuthology.orchestra.run.trial024.stdout:{ 2026-02-17T15:14:30.382 INFO:teuthology.orchestra.run.trial024.stdout: "ops": [], 2026-02-17T15:14:30.382 INFO:teuthology.orchestra.run.trial024.stdout: "num_ops": 0 2026-02-17T15:14:30.382 INFO:teuthology.orchestra.run.trial024.stdout:} 2026-02-17T15:14:30.389 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.1 to be up 2026-02-17T15:14:30.389 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:14:30.539 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:30.539 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":24,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:14:27.323571+0000","last_up_change":"2026-02-17T15:14:26.320849+0000","last_in_change":"2026-02-17T15:13:48.171454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-17T15:14:06.508581+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":4268245710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":4268245710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":4268245710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":4268245710}]},"public_addr":"10.20.193.24:6809/4268245710","cluster_addr":"10.20.193.24:6810/4268245710","heartbeat_back_addr":"10.20.193.24:6812/4268245710","heartbeat_front_addr":"10.20.193.24:6811/4268245710","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":19,"up_from":23,"up_thru":23,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":2457068227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":2457068227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":2457068227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":2457068227}]},"public_addr":"10.20.193.24:6805/2457068227","cluster_addr":"10.20.193.24:6806/2457068227","heartbeat_back_addr":"10.20.193.24:6808/2457068227","heartbeat_front_addr":"10.20.193.24:6807/2457068227","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":1012255776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":1012255776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":1012255776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":1012255776}]},"public_addr":"10.20.193.24:6813/1012255776","cluster_addr":"10.20.193.24:6814/1012255776","heartbeat_back_addr":"10.20.193.24:6816/1012255776","heartbeat_front_addr":"10.20.193.24:6815/1012255776","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":3626123643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":3626123643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":3626123643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":3626123643}]},"public_addr":"10.20.193.24:6801/3626123643","cluster_addr":"10.20.193.24:6802/3626123643","heartbeat_back_addr":"10.20.193.24:6804/3626123643","heartbeat_front_addr":"10.20.193.24:6803/3626123643","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:14:30.551 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.1 is up 2026-02-17T15:14:30.552 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 2.0 2026-02-17T15:14:30.701 INFO:teuthology.orchestra.run.trial024.stderr:instructing pg 2.0s0 on osd.3 to deep-scrub 2026-02-17T15:14:30.711 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 deep-scrub complete (check_time_now=1771341270) 2026-02-17T15:14:30.715 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:14:30.709+0000 7f00da936640 -1 log_channel(cluster) log [ERR] : 2.0 shard 1(1) soid 2:ad551702:::test:head : candidate had a read error, candidate had a missing hinfo key 2026-02-17T15:14:30.716 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:14:30.709+0000 7f00da936640 -1 log_channel(cluster) log [ERR] : 2.0s0 deep-scrub 0 missing, 1 inconsistent objects 2026-02-17T15:14:30.716 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:14:30.709+0000 7f00da936640 -1 log_channel(cluster) log [ERR] : 2.0 deep-scrub 1 errors 2026-02-17T15:14:33.550 INFO:tasks.ceph.mon.a.trial024.stderr:2026-02-17T15:14:33.545+0000 7fc1d094f640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 scrub errors (OSD_SCRUB_ERRORS) 2026-02-17T15:14:33.551 INFO:tasks.ceph.mon.a.trial024.stderr:2026-02-17T15:14:33.545+0000 7fc1d094f640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 1 pg inconsistent (PG_DAMAGED) 2026-02-17T15:14:35.712 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:14:35.713 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:14:35.714 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:14:35.714 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:14:35.799 INFO:teuthology.orchestra.run.trial024.stdout:38654705675 2026-02-17T15:14:35.799 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:14:35.800 INFO:teuthology.orchestra.run.trial024.stdout:38654705675 2026-02-17T15:14:35.801 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:14:35.805 INFO:teuthology.orchestra.run.trial024.stdout:38654705676 2026-02-17T15:14:35.806 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:14:35.809 INFO:teuthology.orchestra.run.trial024.stdout:98784247812 2026-02-17T15:14:35.810 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:14:35.951 INFO:teuthology.orchestra.run.trial024.stdout:38654705674 2026-02-17T15:14:35.955 INFO:teuthology.orchestra.run.trial024.stdout:38654705674 2026-02-17T15:14:35.964 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705675 got 38654705674 for osd.2 2026-02-17T15:14:35.968 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705675 got 38654705674 for osd.3 2026-02-17T15:14:35.996 INFO:teuthology.orchestra.run.trial024.stdout:38654705675 2026-02-17T15:14:36.007 INFO:teuthology.orchestra.run.trial024.stdout:98784247811 2026-02-17T15:14:36.008 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705676 got 38654705675 for osd.0 2026-02-17T15:14:36.018 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247812 got 98784247811 for osd.1 2026-02-17T15:14:36.965 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:14:36.969 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:14:37.008 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:14:37.019 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:14:37.122 INFO:teuthology.orchestra.run.trial024.stdout:38654705674 2026-02-17T15:14:37.135 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705675 got 38654705674 for osd.2 2026-02-17T15:14:37.157 INFO:teuthology.orchestra.run.trial024.stdout:38654705675 2026-02-17T15:14:37.157 INFO:teuthology.orchestra.run.trial024.stdout:38654705674 2026-02-17T15:14:37.169 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705676 got 38654705675 for osd.0 2026-02-17T15:14:37.170 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705675 got 38654705674 for osd.3 2026-02-17T15:14:37.200 INFO:teuthology.orchestra.run.trial024.stdout:98784247811 2026-02-17T15:14:37.211 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247812 got 98784247811 for osd.1 2026-02-17T15:14:38.136 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:14:38.170 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:14:38.170 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:14:38.212 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:14:38.284 INFO:teuthology.orchestra.run.trial024.stdout:38654705676 2026-02-17T15:14:38.297 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705675 got 38654705676 for osd.2 2026-02-17T15:14:38.297 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:38.327 INFO:teuthology.orchestra.run.trial024.stdout:38654705676 2026-02-17T15:14:38.330 INFO:teuthology.orchestra.run.trial024.stdout:38654705677 2026-02-17T15:14:38.339 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705675 got 38654705676 for osd.3 2026-02-17T15:14:38.340 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:38.342 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705676 got 38654705677 for osd.0 2026-02-17T15:14:38.343 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:38.367 INFO:teuthology.orchestra.run.trial024.stdout:98784247812 2026-02-17T15:14:38.379 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247812 got 98784247812 for osd.1 2026-02-17T15:14:38.379 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:38.379 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:14:38.573 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:38.573 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:14:38.584 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2026-02-17T15:14:37.467318+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":344850,"num_objects":4,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":39,"num_write_kb":340,"num_scrub_errors":1,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":1,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":113468,"kb_used_data":872,"kb_used_omap":8,"kb_used_meta":112503,"kb_avail":377373892,"statfs":{"total":386547056640,"available":386430865408,"internally_reserved":0,"allocated":892928,"data_stored":979908,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":9174,"internal_metadata":115203114},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":1,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":1,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.142509"},"pg_stats":[{"pgid":"2.0","version":"19'2","reported_seq":46,"reported_epoch":24,"state":"active+clean+inconsistent","last_fresh":"2026-02-17T15:14:30.713749+0000","last_change":"2026-02-17T15:14:30.713786+0000","last_active":"2026-02-17T15:14:30.713749+0000","last_peered":"2026-02-17T15:14:30.713749+0000","last_clean":"2026-02-17T15:14:30.713749+0000","last_became_active":"2026-02-17T15:14:27.327861+0000","last_became_peered":"2026-02-17T15:14:27.327861+0000","last_unstale":"2026-02-17T15:14:30.713749+0000","last_undegraded":"2026-02-17T15:14:30.713749+0000","last_fullsized":"2026-02-17T15:14:30.713749+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:30.713690+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:30.713690+0000","last_clean_scrub_stamp":"2026-02-17T15:14:07.503000+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T16:17:30.574079+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16658,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":18,"num_scrub_errors":1,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":1,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-17T15:14:27.327944+0000","last_change":"2026-02-17T15:14:27.327944+0000","last_active":"2026-02-17T15:14:27.327944+0000","last_peered":"2026-02-17T15:14:27.327944+0000","last_clean":"2026-02-17T15:14:27.327944+0000","last_became_active":"2026-02-17T15:14:27.327718+0000","last_became_peered":"2026-02-17T15:14:27.327718+0000","last_unstale":"2026-02-17T15:14:27.327944+0000","last_undegraded":"2026-02-17T15:14:27.327944+0000","last_fullsized":"2026-02-17T15:14:27.327944+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T15:37:52.988522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":16658,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":18,"num_scrub_errors":1,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":1,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":49152,"data_compressed":16028,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247812,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27780,"kb_used_data":232,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344060,"statfs":{"total":96636764160,"available":96608317440,"internally_reserved":0,"allocated":237568,"data_stored":413169,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":28176086},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705676,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31328,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340512,"statfs":{"total":96636764160,"available":96604684288,"internally_reserved":0,"allocated":212992,"data_stored":84977,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705676,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27172,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344668,"statfs":{"total":96636764160,"available":96608940032,"internally_reserved":0,"allocated":212992,"data_stored":84977,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705677,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":229376,"data_stored":396785,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16384,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16384,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16384,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T15:14:38.585 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "19'2", 'reported_seq': 46, 'reported_epoch': 24, 'state': 'active+clean+inconsistent', 'last_fresh': '2026-02-17T15:14:30.713749+0000', 'last_change': '2026-02-17T15:14:30.713786+0000', 'last_active': '2026-02-17T15:14:30.713749+0000', 'last_peered': '2026-02-17T15:14:30.713749+0000', 'last_clean': '2026-02-17T15:14:30.713749+0000', 'last_became_active': '2026-02-17T15:14:27.327861+0000', 'last_became_peered': '2026-02-17T15:14:27.327861+0000', 'last_unstale': '2026-02-17T15:14:30.713749+0000', 'last_undegraded': '2026-02-17T15:14:30.713749+0000', 'last_fullsized': '2026-02-17T15:14:30.713749+0000', 'mapping_epoch': 23, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 24, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:30.713690+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:30.713690+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:07.503000+0000', 'objects_scrubbed': 2, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 1, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-18T16:17:30.574079+0000', 'scrub_duration': 3, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16658, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 18, 'num_scrub_errors': 1, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 1, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:14:38.586 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg repair 2.0 2026-02-17T15:14:38.736 INFO:teuthology.orchestra.run.trial024.stderr:instructing pg 2.0s0 on osd.3 to repair 2026-02-17T15:14:38.747 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 deep-scrub complete (check_time_now=1771341278) 2026-02-17T15:14:39.587 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:14:39.577+0000 7f00da936640 -1 log_channel(cluster) log [ERR] : 2.0 shard 1(1) soid 2:ad551702:::test:head : candidate had a read error, candidate had a missing hinfo key 2026-02-17T15:14:39.587 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:14:39.577+0000 7f00da936640 -1 log_channel(cluster) log [ERR] : 2.0s0 repair 0 missing, 1 inconsistent objects 2026-02-17T15:14:39.587 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:14:39.577+0000 7f00da936640 -1 log_channel(cluster) log [ERR] : 2.0s0 repair 0 missing, 1 inconsistent objects 2026-02-17T15:14:39.587 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:14:39.577+0000 7f00da936640 -1 log_channel(cluster) log [ERR] : 2.0 repair 1 errors, 1 fixed 2026-02-17T15:14:43.747 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:14:43.748 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:14:43.749 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:14:43.749 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:14:43.835 INFO:teuthology.orchestra.run.trial024.stdout:38654705678 2026-02-17T15:14:43.836 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:14:43.837 INFO:teuthology.orchestra.run.trial024.stdout:38654705679 2026-02-17T15:14:43.838 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:14:43.839 INFO:teuthology.orchestra.run.trial024.stdout:38654705678 2026-02-17T15:14:43.839 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:14:43.845 INFO:teuthology.orchestra.run.trial024.stdout:98784247814 2026-02-17T15:14:43.846 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:14:43.999 INFO:teuthology.orchestra.run.trial024.stdout:38654705677 2026-02-17T15:14:44.012 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705678 got 38654705677 for osd.2 2026-02-17T15:14:44.033 INFO:teuthology.orchestra.run.trial024.stdout:38654705677 2026-02-17T15:14:44.033 INFO:teuthology.orchestra.run.trial024.stdout:98784247813 2026-02-17T15:14:44.041 INFO:teuthology.orchestra.run.trial024.stdout:38654705678 2026-02-17T15:14:44.045 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705678 got 38654705677 for osd.3 2026-02-17T15:14:44.045 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247814 got 98784247813 for osd.1 2026-02-17T15:14:44.054 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705679 got 38654705678 for osd.0 2026-02-17T15:14:45.013 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:14:45.046 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:14:45.046 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:14:45.055 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:14:45.162 INFO:teuthology.orchestra.run.trial024.stdout:38654705677 2026-02-17T15:14:45.175 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705678 got 38654705677 for osd.2 2026-02-17T15:14:45.204 INFO:teuthology.orchestra.run.trial024.stdout:38654705678 2026-02-17T15:14:45.205 INFO:teuthology.orchestra.run.trial024.stdout:98784247813 2026-02-17T15:14:45.207 INFO:teuthology.orchestra.run.trial024.stdout:38654705677 2026-02-17T15:14:45.216 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705679 got 38654705678 for osd.0 2026-02-17T15:14:45.217 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247814 got 98784247813 for osd.1 2026-02-17T15:14:45.219 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705678 got 38654705677 for osd.3 2026-02-17T15:14:46.176 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:14:46.217 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:14:46.218 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:14:46.220 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:14:46.328 INFO:teuthology.orchestra.run.trial024.stdout:38654705678 2026-02-17T15:14:46.341 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705678 got 38654705678 for osd.2 2026-02-17T15:14:46.342 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:46.362 INFO:teuthology.orchestra.run.trial024.stdout:98784247815 2026-02-17T15:14:46.363 INFO:teuthology.orchestra.run.trial024.stdout:38654705679 2026-02-17T15:14:46.375 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247814 got 98784247815 for osd.1 2026-02-17T15:14:46.375 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:46.376 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705679 got 38654705679 for osd.0 2026-02-17T15:14:46.376 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:46.412 INFO:teuthology.orchestra.run.trial024.stdout:38654705678 2026-02-17T15:14:46.424 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705678 got 38654705678 for osd.3 2026-02-17T15:14:46.424 DEBUG:teuthology.parallel:result is None 2026-02-17T15:14:46.424 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:14:46.609 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:14:46.610 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:14:46.620 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2026-02-17T15:14:45.468327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":344850,"num_objects":4,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":39,"num_write_kb":340,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":113452,"kb_used_data":872,"kb_used_omap":8,"kb_used_meta":112503,"kb_avail":377373908,"statfs":{"total":386547056640,"available":386430881792,"internally_reserved":0,"allocated":892928,"data_stored":979908,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":9174,"internal_metadata":115203114},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":-1,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":-1,"num_objects_recovered":1,"num_bytes_recovered":8329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001492"},"pg_stats":[{"pgid":"2.0","version":"19'2","reported_seq":62,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-17T15:14:39.588641+0000","last_change":"2026-02-17T15:14:39.589001+0000","last_active":"2026-02-17T15:14:39.588641+0000","last_peered":"2026-02-17T15:14:39.588641+0000","last_clean":"2026-02-17T15:14:39.588641+0000","last_became_active":"2026-02-17T15:14:27.327861+0000","last_became_peered":"2026-02-17T15:14:27.327861+0000","last_unstale":"2026-02-17T15:14:39.588641+0000","last_undegraded":"2026-02-17T15:14:39.588641+0000","last_fullsized":"2026-02-17T15:14:39.588641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T20:58:13.255925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16658,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":43,"reported_epoch":24,"state":"active+clean","last_fresh":"2026-02-17T15:14:27.327944+0000","last_change":"2026-02-17T15:14:27.327944+0000","last_active":"2026-02-17T15:14:27.327944+0000","last_peered":"2026-02-17T15:14:27.327944+0000","last_clean":"2026-02-17T15:14:27.327944+0000","last_became_active":"2026-02-17T15:14:27.327718+0000","last_became_peered":"2026-02-17T15:14:27.327718+0000","last_unstale":"2026-02-17T15:14:27.327944+0000","last_undegraded":"2026-02-17T15:14:27.327944+0000","last_fullsized":"2026-02-17T15:14:27.327944+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T15:37:52.988522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":16658,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":8329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":49152,"data_compressed":16028,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":3,"acting":3,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":656384,"data_compressed":1186,"data_compressed_allocated":40960,"data_compressed_original":655360,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247815,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27768,"kb_used_data":232,"kb_used_omap":4,"kb_used_meta":27515,"kb_avail":94344072,"statfs":{"total":96636764160,"available":96608329728,"internally_reserved":0,"allocated":237568,"data_stored":413169,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":28176086},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":9,"seq":38654705678,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31328,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340512,"statfs":{"total":96636764160,"available":96604684288,"internally_reserved":0,"allocated":212992,"data_stored":84977,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705678,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27168,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344672,"statfs":{"total":96636764160,"available":96608944128,"internally_reserved":0,"allocated":212992,"data_stored":84977,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705679,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":26942,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":229376,"data_stored":396785,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":27589062},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16384,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16384,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":16384,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-17T15:14:46.620 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "19'2", 'reported_seq': 62, 'reported_epoch': 24, 'state': 'active+clean', 'last_fresh': '2026-02-17T15:14:39.588641+0000', 'last_change': '2026-02-17T15:14:39.589001+0000', 'last_active': '2026-02-17T15:14:39.588641+0000', 'last_peered': '2026-02-17T15:14:39.588641+0000', 'last_clean': '2026-02-17T15:14:39.588641+0000', 'last_became_active': '2026-02-17T15:14:27.327861+0000', 'last_became_peered': '2026-02-17T15:14:27.327861+0000', 'last_unstale': '2026-02-17T15:14:39.588641+0000', 'last_undegraded': '2026-02-17T15:14:39.588641+0000', 'last_fullsized': '2026-02-17T15:14:39.588641+0000', 'mapping_epoch': 23, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 16, 'last_epoch_clean': 24, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-18T20:58:13.255925+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 16658, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 18, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 8329, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:14:46.621 INFO:tasks.ec_inconsistent_hinfo:remove test object hash info from primary osd.3 shard and test backfill 2026-02-17T15:14:46.621 DEBUG:tasks.ec_inconsistent_hinfo:write some data 2026-02-17T15:14:46.621 INFO:tasks.util.rados:rados -p unique_pool_0 bench 30 write -b 4096 --no-cleanup 2026-02-17T15:14:46.621 DEBUG:teuthology.orchestra.run.trial024:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p unique_pool_0 bench 30 write -b 4096 --no-cleanup 2026-02-17T15:14:46.643 INFO:teuthology.orchestra.run.trial024.stderr:INFO: op_size has been rounded to 8192 2026-02-17T15:14:46.644 INFO:teuthology.orchestra.run.trial024.stdout:hints = 1 2026-02-17T15:14:46.644 INFO:teuthology.orchestra.run.trial024.stdout:Maintaining 16 concurrent writes of 8192 bytes to objects of size 8192 for up to 30 seconds or 0 objects 2026-02-17T15:14:46.644 INFO:teuthology.orchestra.run.trial024.stdout:Object prefix: benchmark_data_trial024_24697 2026-02-17T15:14:46.644 INFO:teuthology.orchestra.run.trial024.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-17T15:14:46.644 INFO:teuthology.orchestra.run.trial024.stdout: 0 0 0 0 0 0 - 0 2026-02-17T15:14:47.644 INFO:teuthology.orchestra.run.trial024.stdout: 1 16 2281 2265 17.6938 17.6953 0.00624643 0.00704012 2026-02-17T15:14:48.644 INFO:teuthology.orchestra.run.trial024.stdout: 2 16 4815 4799 18.7446 19.7969 0.00726972 0.00664716 2026-02-17T15:14:49.644 INFO:teuthology.orchestra.run.trial024.stdout: 3 16 7072 7056 18.3736 17.6328 0.00747847 0.0067941 2026-02-17T15:14:50.644 INFO:teuthology.orchestra.run.trial024.stdout: 4 16 9537 9521 18.5942 19.2578 0.00682587 0.00671506 2026-02-17T15:14:51.644 INFO:teuthology.orchestra.run.trial024.stdout: 5 16 11941 11925 18.6315 18.7812 0.00527162 0.00670412 2026-02-17T15:14:52.645 INFO:teuthology.orchestra.run.trial024.stdout: 6 16 14583 14567 18.9661 20.6406 0.00585607 0.00658604 2026-02-17T15:14:53.645 INFO:teuthology.orchestra.run.trial024.stdout: 7 16 17019 17003 18.9752 19.0312 0.00669649 0.00658282 2026-02-17T15:14:54.645 INFO:teuthology.orchestra.run.trial024.stdout: 8 16 19519 19503 19.0445 19.5312 0.006325 0.00655923 2026-02-17T15:14:55.645 INFO:teuthology.orchestra.run.trial024.stdout: 9 16 21811 21795 18.9179 17.9062 0.0073912 0.00660323 2026-02-17T15:14:56.645 INFO:teuthology.orchestra.run.trial024.stdout: 10 16 24057 24041 18.7806 17.5469 0.00611509 0.00665224 2026-02-17T15:14:57.645 INFO:teuthology.orchestra.run.trial024.stdout: 11 16 26288 26272 18.6577 17.4297 0.00665872 0.00669618 2026-02-17T15:14:58.645 INFO:teuthology.orchestra.run.trial024.stdout: 12 16 28621 28605 18.6217 18.2266 0.0049357 0.00670956 2026-02-17T15:14:59.645 INFO:teuthology.orchestra.run.trial024.stdout: 13 16 31077 31061 18.6651 19.1875 0.00588466 0.00669401 2026-02-17T15:15:00.645 INFO:teuthology.orchestra.run.trial024.stdout: 14 15 33378 33363 18.6164 17.9844 0.00694414 0.00671139 2026-02-17T15:15:01.645 INFO:teuthology.orchestra.run.trial024.stdout: 15 16 35737 35721 18.6033 18.4219 0.00715176 0.00671593 2026-02-17T15:15:02.646 INFO:teuthology.orchestra.run.trial024.stdout: 16 16 38251 38235 18.668 19.6406 0.00585806 0.0066935 2026-02-17T15:15:03.646 INFO:teuthology.orchestra.run.trial024.stdout: 17 16 40393 40377 18.5542 16.7344 0.00770652 0.00673359 2026-02-17T15:15:04.646 INFO:teuthology.orchestra.run.trial024.stdout: 18 16 42779 42763 18.5589 18.6406 0.00666226 0.00673264 2026-02-17T15:15:05.645 INFO:teuthology.orchestra.run.trial024.stdout: 19 16 45207 45191 18.5804 18.9688 0.00668413 0.00672493 2026-02-17T15:15:06.646 INFO:teuthology.orchestra.run.trial024.stdout:2026-02-17T15:15:06.643224+0000 min lat: 0.001978 max lat: 0.0134661 avg lat: 0.00672248 2026-02-17T15:15:06.646 INFO:teuthology.orchestra.run.trial024.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-17T15:15:06.646 INFO:teuthology.orchestra.run.trial024.stdout: 20 16 47603 47587 18.5873 18.7188 0.00791668 0.00672248 2026-02-17T15:15:07.646 INFO:teuthology.orchestra.run.trial024.stdout: 21 16 50001 49985 18.5942 18.7344 0.00606332 0.00672003 2026-02-17T15:15:08.646 INFO:teuthology.orchestra.run.trial024.stdout: 22 16 52399 52383 18.6005 18.7344 0.00572258 0.00671776 2026-02-17T15:15:09.646 INFO:teuthology.orchestra.run.trial024.stdout: 23 16 54923 54907 18.6491 19.7188 0.00651462 0.00670035 2026-02-17T15:15:10.646 INFO:teuthology.orchestra.run.trial024.stdout: 24 16 57223 57207 18.6207 17.9688 0.00643745 0.00671052 2026-02-17T15:15:11.646 INFO:teuthology.orchestra.run.trial024.stdout: 25 16 59582 59566 18.613 18.4297 0.00708297 0.00671316 2026-02-17T15:15:12.646 INFO:teuthology.orchestra.run.trial024.stdout: 26 16 61716 61700 18.5383 16.6719 0.00623695 0.00674038 2026-02-17T15:15:13.646 INFO:teuthology.orchestra.run.trial024.stdout: 27 16 64119 64103 18.5469 18.7734 0.00676869 0.00673719 2026-02-17T15:15:14.647 INFO:teuthology.orchestra.run.trial024.stdout: 28 16 66581 66565 18.5714 19.2344 0.00644669 0.00672846 2026-02-17T15:15:15.647 INFO:teuthology.orchestra.run.trial024.stdout: 29 16 68919 68903 18.5609 18.2656 0.00598253 0.00673236 2026-02-17T15:15:16.647 INFO:teuthology.orchestra.run.trial024.stdout: 30 9 71526 71517 18.6228 20.4219 0.00542707 0.00671004 2026-02-17T15:15:17.647 INFO:teuthology.orchestra.run.trial024.stdout:Total time run: 30.0048 2026-02-17T15:15:17.647 INFO:teuthology.orchestra.run.trial024.stdout:Total writes made: 71526 2026-02-17T15:15:17.647 INFO:teuthology.orchestra.run.trial024.stdout:Write size: 8192 2026-02-17T15:15:17.647 INFO:teuthology.orchestra.run.trial024.stdout:Object size: 8192 2026-02-17T15:15:17.647 INFO:teuthology.orchestra.run.trial024.stdout:Bandwidth (MB/sec): 18.6236 2026-02-17T15:15:17.648 INFO:teuthology.orchestra.run.trial024.stdout:Stddev Bandwidth: 0.959861 2026-02-17T15:15:17.648 INFO:teuthology.orchestra.run.trial024.stdout:Max bandwidth (MB/sec): 20.6406 2026-02-17T15:15:17.648 INFO:teuthology.orchestra.run.trial024.stdout:Min bandwidth (MB/sec): 16.6719 2026-02-17T15:15:17.648 INFO:teuthology.orchestra.run.trial024.stdout:Average IOPS: 2383 2026-02-17T15:15:17.648 INFO:teuthology.orchestra.run.trial024.stdout:Stddev IOPS: 122.862 2026-02-17T15:15:17.648 INFO:teuthology.orchestra.run.trial024.stdout:Max IOPS: 2642 2026-02-17T15:15:17.648 INFO:teuthology.orchestra.run.trial024.stdout:Min IOPS: 2134 2026-02-17T15:15:17.648 INFO:teuthology.orchestra.run.trial024.stdout:Average Latency(s): 0.0067099 2026-02-17T15:15:17.648 INFO:teuthology.orchestra.run.trial024.stdout:Stddev Latency(s): 0.00107129 2026-02-17T15:15:17.648 INFO:teuthology.orchestra.run.trial024.stdout:Max latency(s): 0.0134661 2026-02-17T15:15:17.648 INFO:teuthology.orchestra.run.trial024.stdout:Min latency(s): 0.001978 2026-02-17T15:15:17.653 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:15:17.844 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:15:17.845 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":26,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:14:50.258524+0000","last_up_change":"2026-02-17T15:14:26.320849+0000","last_in_change":"2026-02-17T15:13:48.171454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-17T15:14:06.508581+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":4268245710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":4268245710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":4268245710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":4268245710}]},"public_addr":"10.20.193.24:6809/4268245710","cluster_addr":"10.20.193.24:6810/4268245710","heartbeat_back_addr":"10.20.193.24:6812/4268245710","heartbeat_front_addr":"10.20.193.24:6811/4268245710","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":19,"up_from":23,"up_thru":23,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":2457068227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":2457068227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":2457068227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":2457068227}]},"public_addr":"10.20.193.24:6805/2457068227","cluster_addr":"10.20.193.24:6806/2457068227","heartbeat_back_addr":"10.20.193.24:6808/2457068227","heartbeat_front_addr":"10.20.193.24:6807/2457068227","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":1012255776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":1012255776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":1012255776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":1012255776}]},"public_addr":"10.20.193.24:6813/1012255776","cluster_addr":"10.20.193.24:6814/1012255776","heartbeat_back_addr":"10.20.193.24:6816/1012255776","heartbeat_front_addr":"10.20.193.24:6815/1012255776","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":3626123643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":3626123643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":3626123643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":3626123643}]},"public_addr":"10.20.193.24:6801/3626123643","cluster_addr":"10.20.193.24:6802/3626123643","heartbeat_back_addr":"10.20.193.24:6804/3626123643","heartbeat_front_addr":"10.20.193.24:6803/3626123643","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.138188+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.488216+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.278612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:54.537485+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:15:17.855 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-02-17T15:15:18.015 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:15:18.026 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":26,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,1,2],"up_primary":3,"acting":[3,1,2],"acting_primary":3} 2026-02-17T15:15:18.027 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-17T15:15:18.027 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T15:15:18.107 INFO:tasks.ceph.osd.3:Stopped 2026-02-17T15:15:18.107 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 "$object" rm-attr hinfo_key 2026-02-17T15:15:18.107 DEBUG:teuthology.orchestra.run.trial024:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list |grep '"'"'"oid":"test"'"'"') 2026-02-17T15:15:18.108 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 "$object" rm-attr hinfo_key' 2026-02-17T15:15:18.111 INFO:teuthology.orchestra.run.trial024.stderr:+ '[' -z '' ']' 2026-02-17T15:15:18.112 INFO:teuthology.orchestra.run.trial024.stderr:+ return 2026-02-17T15:15:18.112 INFO:teuthology.orchestra.run.trial024.stderr:+ case $- in 2026-02-17T15:15:18.112 INFO:teuthology.orchestra.run.trial024.stderr:+ return 2026-02-17T15:15:18.113 INFO:teuthology.orchestra.run.trial024.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list 2026-02-17T15:15:18.113 INFO:teuthology.orchestra.run.trial024.stderr:++ grep '"oid":"test"' 2026-02-17T15:15:19.154 INFO:teuthology.orchestra.run.trial024.stderr:Error getting attr on : 2.0s0_head,0#-4:00000000:::scrub_2.0s0:head#, (61) No data available 2026-02-17T15:15:20.713 INFO:teuthology.orchestra.run.trial024.stderr:+ object='["2.0s0",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":0,"max":0}]' 2026-02-17T15:15:20.713 INFO:teuthology.orchestra.run.trial024.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 '["2.0s0",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":0,"max":0}]' rm-attr hinfo_key 2026-02-17T15:15:26.470 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-17T15:15:26.471 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2026-02-17T15:15:26.513 INFO:tasks.ceph.osd.3:Started 2026-02-17T15:15:26.513 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-02-17T15:15:26.572 INFO:teuthology.orchestra.run.trial024.stderr:no valid command found; 10 closest matches: 2026-02-17T15:15:26.572 INFO:teuthology.orchestra.run.trial024.stderr:0 2026-02-17T15:15:26.572 INFO:teuthology.orchestra.run.trial024.stderr:1 2026-02-17T15:15:26.572 INFO:teuthology.orchestra.run.trial024.stderr:2 2026-02-17T15:15:26.572 INFO:teuthology.orchestra.run.trial024.stderr:abort 2026-02-17T15:15:26.572 INFO:teuthology.orchestra.run.trial024.stderr:assert 2026-02-17T15:15:26.572 INFO:teuthology.orchestra.run.trial024.stderr:config diff 2026-02-17T15:15:26.572 INFO:teuthology.orchestra.run.trial024.stderr:config diff get 2026-02-17T15:15:26.573 INFO:teuthology.orchestra.run.trial024.stderr:config get 2026-02-17T15:15:26.573 INFO:teuthology.orchestra.run.trial024.stderr:config help [] 2026-02-17T15:15:26.573 INFO:teuthology.orchestra.run.trial024.stderr:config set ... 2026-02-17T15:15:26.573 INFO:teuthology.orchestra.run.trial024.stderr:admin_socket: invalid command 2026-02-17T15:15:26.574 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-17T15:15:26.574 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2026-02-17T15:15:27.801 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:15:27.795+0000 7f24ad2668c0 -1 Falling back to public interface 2026-02-17T15:15:31.575 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-02-17T15:15:31.643 INFO:teuthology.orchestra.run.trial024.stderr:no valid command found; 10 closest matches: 2026-02-17T15:15:31.643 INFO:teuthology.orchestra.run.trial024.stderr:0 2026-02-17T15:15:31.643 INFO:teuthology.orchestra.run.trial024.stderr:1 2026-02-17T15:15:31.643 INFO:teuthology.orchestra.run.trial024.stderr:2 2026-02-17T15:15:31.643 INFO:teuthology.orchestra.run.trial024.stderr:abort 2026-02-17T15:15:31.644 INFO:teuthology.orchestra.run.trial024.stderr:assert 2026-02-17T15:15:31.644 INFO:teuthology.orchestra.run.trial024.stderr:bluestore allocator dump block 2026-02-17T15:15:31.644 INFO:teuthology.orchestra.run.trial024.stderr:bluestore allocator fragmentation block 2026-02-17T15:15:31.644 INFO:teuthology.orchestra.run.trial024.stderr:bluestore allocator fragmentation histogram block [] [] 2026-02-17T15:15:31.644 INFO:teuthology.orchestra.run.trial024.stderr:bluestore allocator score block 2026-02-17T15:15:31.644 INFO:teuthology.orchestra.run.trial024.stderr:config diff 2026-02-17T15:15:31.644 INFO:teuthology.orchestra.run.trial024.stderr:admin_socket: invalid command 2026-02-17T15:15:31.645 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-17T15:15:31.646 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2026-02-17T15:15:31.961 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:15:31.955+0000 7f24ad2668c0 -1 osd.3 26 log_to_monitors true 2026-02-17T15:15:32.563 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:15:32.555+0000 7f24a504a640 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T15:15:36.647 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-02-17T15:15:36.719 INFO:teuthology.orchestra.run.trial024.stdout:{ 2026-02-17T15:15:36.719 INFO:teuthology.orchestra.run.trial024.stdout: "ops": [], 2026-02-17T15:15:36.719 INFO:teuthology.orchestra.run.trial024.stdout: "num_ops": 0 2026-02-17T15:15:36.720 INFO:teuthology.orchestra.run.trial024.stdout:} 2026-02-17T15:15:36.726 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.3 to be up 2026-02-17T15:15:36.727 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:15:36.879 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:15:36.880 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":31,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:15:34.558330+0000","last_up_change":"2026-02-17T15:15:33.556007+0000","last_in_change":"2026-02-17T15:13:48.171454+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-17T15:14:06.508581+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":4268245710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":4268245710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":4268245710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":4268245710}]},"public_addr":"10.20.193.24:6809/4268245710","cluster_addr":"10.20.193.24:6810/4268245710","heartbeat_back_addr":"10.20.193.24:6812/4268245710","heartbeat_front_addr":"10.20.193.24:6811/4268245710","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":19,"up_from":23,"up_thru":27,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":2457068227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":2457068227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":2457068227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":2457068227}]},"public_addr":"10.20.193.24:6805/2457068227","cluster_addr":"10.20.193.24:6806/2457068227","heartbeat_back_addr":"10.20.193.24:6808/2457068227","heartbeat_front_addr":"10.20.193.24:6807/2457068227","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":1012255776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":1012255776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":1012255776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":1012255776}]},"public_addr":"10.20.193.24:6813/1012255776","cluster_addr":"10.20.193.24:6814/1012255776","heartbeat_back_addr":"10.20.193.24:6816/1012255776","heartbeat_front_addr":"10.20.193.24:6815/1012255776","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":26,"up_from":30,"up_thru":30,"down_at":27,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":306838151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":306838151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":306838151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":306838151}]},"public_addr":"10.20.193.24:6801/306838151","cluster_addr":"10.20.193.24:6802/306838151","heartbeat_back_addr":"10.20.193.24:6804/306838151","heartbeat_front_addr":"10.20.193.24:6803/306838151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.138188+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.488216+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.278612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:54.537485+0000","dead_epoch":27}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:15:36.890 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.3 is up 2026-02-17T15:15:36.891 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2026-02-17T15:15:37.038 INFO:tasks.ceph.mon.a.trial024.stderr:2026-02-17T15:15:37.031+0000 7fc1ce14a640 -1 mon.a@0(leader).osd e31 definitely_dead 0 2026-02-17T15:15:37.935 INFO:teuthology.orchestra.run.trial024.stderr:marked out osd.1. 2026-02-17T15:15:37.949 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 backfill from osd.1 to osd.0 complete 2026-02-17T15:15:42.951 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:15:42.951 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:15:42.952 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:15:42.952 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:15:43.040 INFO:teuthology.orchestra.run.trial024.stdout:38654705692 2026-02-17T15:15:43.041 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:15:43.042 INFO:teuthology.orchestra.run.trial024.stdout:98784247827 2026-02-17T15:15:43.042 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:15:43.043 INFO:teuthology.orchestra.run.trial024.stdout:38654705691 2026-02-17T15:15:43.044 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:15:43.047 INFO:teuthology.orchestra.run.trial024.stdout:128849018884 2026-02-17T15:15:43.048 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:15:43.215 INFO:teuthology.orchestra.run.trial024.stdout:38654705690 2026-02-17T15:15:43.229 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705692 got 38654705690 for osd.0 2026-02-17T15:15:43.240 INFO:teuthology.orchestra.run.trial024.stdout:98784247826 2026-02-17T15:15:43.249 INFO:teuthology.orchestra.run.trial024.stdout:38654705689 2026-02-17T15:15:43.253 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247827 got 98784247826 for osd.1 2026-02-17T15:15:43.259 INFO:teuthology.orchestra.run.trial024.stdout:128849018882 2026-02-17T15:15:43.261 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705691 got 38654705689 for osd.2 2026-02-17T15:15:43.272 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018884 got 128849018882 for osd.3 2026-02-17T15:15:44.229 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:15:44.254 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:15:44.263 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:15:44.273 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:15:44.395 INFO:teuthology.orchestra.run.trial024.stdout:38654705692 2026-02-17T15:15:44.408 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705692 got 38654705692 for osd.0 2026-02-17T15:15:44.409 DEBUG:teuthology.parallel:result is None 2026-02-17T15:15:44.426 INFO:teuthology.orchestra.run.trial024.stdout:98784247827 2026-02-17T15:15:44.428 INFO:teuthology.orchestra.run.trial024.stdout:38654705691 2026-02-17T15:15:44.436 INFO:teuthology.orchestra.run.trial024.stdout:128849018884 2026-02-17T15:15:44.439 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247827 got 98784247827 for osd.1 2026-02-17T15:15:44.440 DEBUG:teuthology.parallel:result is None 2026-02-17T15:15:44.441 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705691 got 38654705691 for osd.2 2026-02-17T15:15:44.441 DEBUG:teuthology.parallel:result is None 2026-02-17T15:15:44.450 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018884 got 128849018884 for osd.3 2026-02-17T15:15:44.450 DEBUG:teuthology.parallel:result is None 2026-02-17T15:15:44.451 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:15:44.655 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:15:44.656 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:15:44.667 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":85,"stamp":"2026-02-17T15:15:43.476028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":69599,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1934,"num_bytes_recovered":16155273,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":832860,"kb_used_data":578584,"kb_used_omap":7,"kb_used_meta":254200,"kb_avail":282282660,"statfs":{"total":289910292480,"available":289057443840,"internally_reserved":0,"allocated":592470016,"data_stored":592550433,"data_compressed":12954,"data_compressed_allocated":57344,"data_compressed_original":688128,"omap_allocated":7634,"internal_metadata":260301358},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-71529,"num_objects_misplaced":69599,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1933,"num_bytes_recovered":16146944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000807"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":75480,"reported_epoch":34,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:15:43.033252+0000","last_change":"2026-02-17T15:15:40.196063+0000","last_active":"2026-02-17T15:15:43.033252+0000","last_peered":"2026-02-17T15:15:43.033252+0000","last_clean":"2026-02-17T15:15:34.564274+0000","last_became_active":"2026-02-17T15:15:40.195294+0000","last_became_peered":"2026-02-17T15:15:40.195294+0000","last_unstale":"2026-02-17T15:15:43.033252+0000","last_undegraded":"2026-02-17T15:15:43.033252+0000","last_fullsized":"2026-02-17T15:15:43.033252+0000","mapping_epoch":33,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":69599,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":15827081,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-17T15:15:40.180180+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:15:40.180180+0000","last_peered":"2026-02-17T15:15:40.180180+0000","last_clean":"2026-02-17T15:15:40.180180+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:15:40.180180+0000","last_undegraded":"2026-02-17T15:15:40.180180+0000","last_fullsized":"2026-02-17T15:15:40.180180+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":69599,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1932,"num_bytes_recovered":15827081,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":886833152,"data_stored":886857728,"data_compressed":16028,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247827,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018884,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325720,"kb_used_data":286396,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046120,"statfs":{"total":96636764160,"available":96303226880,"internally_reserved":0,"allocated":293269504,"data_stored":293095435,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466084,"kb_used_data":286416,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905756,"statfs":{"total":96636764160,"available":96159494144,"internally_reserved":0,"allocated":293289984,"data_stored":293423115,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705692,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":5772,"kb_used_omap":1,"kb_used_meta":35262,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":5910528,"data_stored":6031883,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":36108742},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":7888896,"data_stored":7888896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:15:44.668 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 75480, 'reported_epoch': 34, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:15:43.033252+0000', 'last_change': '2026-02-17T15:15:40.196063+0000', 'last_active': '2026-02-17T15:15:43.033252+0000', 'last_peered': '2026-02-17T15:15:43.033252+0000', 'last_clean': '2026-02-17T15:15:34.564274+0000', 'last_became_active': '2026-02-17T15:15:40.195294+0000', 'last_became_peered': '2026-02-17T15:15:40.195294+0000', 'last_unstale': '2026-02-17T15:15:43.033252+0000', 'last_undegraded': '2026-02-17T15:15:43.033252+0000', 'last_fullsized': '2026-02-17T15:15:43.033252+0000', 'mapping_epoch': 33, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 69599, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1932, 'num_bytes_recovered': 15827081, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:15:44.668 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:15:49.669 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:15:49.670 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:15:49.671 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:15:49.671 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:15:49.761 INFO:teuthology.orchestra.run.trial024.stdout:128849018886 2026-02-17T15:15:49.761 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:15:49.762 INFO:teuthology.orchestra.run.trial024.stdout:98784247829 2026-02-17T15:15:49.763 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:15:49.763 INFO:teuthology.orchestra.run.trial024.stdout:38654705694 2026-02-17T15:15:49.764 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:15:49.765 INFO:teuthology.orchestra.run.trial024.stdout:38654705693 2026-02-17T15:15:49.765 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:15:49.956 INFO:teuthology.orchestra.run.trial024.stdout:38654705692 2026-02-17T15:15:49.957 INFO:teuthology.orchestra.run.trial024.stdout:128849018885 2026-02-17T15:15:49.960 INFO:teuthology.orchestra.run.trial024.stdout:98784247828 2026-02-17T15:15:49.965 INFO:teuthology.orchestra.run.trial024.stdout:38654705693 2026-02-17T15:15:49.969 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705693 got 38654705692 for osd.2 2026-02-17T15:15:49.970 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018886 got 128849018885 for osd.3 2026-02-17T15:15:49.973 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247829 got 98784247828 for osd.1 2026-02-17T15:15:49.978 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705694 got 38654705693 for osd.0 2026-02-17T15:15:50.971 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:15:50.971 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:15:50.974 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:15:50.978 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:15:51.123 INFO:teuthology.orchestra.run.trial024.stdout:38654705692 2026-02-17T15:15:51.127 INFO:teuthology.orchestra.run.trial024.stdout:128849018885 2026-02-17T15:15:51.136 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705693 got 38654705692 for osd.2 2026-02-17T15:15:51.140 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018886 got 128849018885 for osd.3 2026-02-17T15:15:51.177 INFO:teuthology.orchestra.run.trial024.stdout:38654705693 2026-02-17T15:15:51.180 INFO:teuthology.orchestra.run.trial024.stdout:98784247828 2026-02-17T15:15:51.190 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705694 got 38654705693 for osd.0 2026-02-17T15:15:51.193 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247829 got 98784247828 for osd.1 2026-02-17T15:15:52.137 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:15:52.140 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:15:52.191 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:15:52.194 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:15:52.311 INFO:teuthology.orchestra.run.trial024.stdout:38654705693 2026-02-17T15:15:52.324 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705693 got 38654705693 for osd.2 2026-02-17T15:15:52.326 DEBUG:teuthology.parallel:result is None 2026-02-17T15:15:52.339 INFO:teuthology.orchestra.run.trial024.stdout:128849018886 2026-02-17T15:15:52.352 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018886 got 128849018886 for osd.3 2026-02-17T15:15:52.352 DEBUG:teuthology.parallel:result is None 2026-02-17T15:15:52.385 INFO:teuthology.orchestra.run.trial024.stdout:98784247830 2026-02-17T15:15:52.386 INFO:teuthology.orchestra.run.trial024.stdout:38654705694 2026-02-17T15:15:52.397 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247829 got 98784247830 for osd.1 2026-02-17T15:15:52.397 DEBUG:teuthology.parallel:result is None 2026-02-17T15:15:52.398 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705694 got 38654705694 for osd.0 2026-02-17T15:15:52.398 DEBUG:teuthology.parallel:result is None 2026-02-17T15:15:52.399 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:15:52.590 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:15:52.590 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:15:52.601 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":89,"stamp":"2026-02-17T15:15:51.476981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":64661,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6872,"num_bytes_recovered":56607369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":867332,"kb_used_data":599172,"kb_used_omap":7,"kb_used_meta":268088,"kb_avail":282248188,"statfs":{"total":289910292480,"available":289022144512,"internally_reserved":0,"allocated":613552128,"data_stored":613632545,"data_compressed":12954,"data_compressed_allocated":57344,"data_compressed_original":688128,"omap_allocated":7634,"internal_metadata":274522670},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":64661,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6871,"num_bytes_recovered":56599040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.540565"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":85356,"reported_epoch":34,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:15:49.745400+0000","last_change":"2026-02-17T15:15:40.196063+0000","last_active":"2026-02-17T15:15:49.745400+0000","last_peered":"2026-02-17T15:15:49.745400+0000","last_clean":"2026-02-17T15:15:34.564274+0000","last_became_active":"2026-02-17T15:15:40.195294+0000","last_became_peered":"2026-02-17T15:15:40.195294+0000","last_unstale":"2026-02-17T15:15:49.745400+0000","last_undegraded":"2026-02-17T15:15:49.745400+0000","last_fullsized":"2026-02-17T15:15:49.745400+0000","mapping_epoch":33,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":64661,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6870,"num_bytes_recovered":56279177,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-17T15:15:40.180180+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:15:40.180180+0000","last_peered":"2026-02-17T15:15:40.180180+0000","last_clean":"2026-02-17T15:15:40.180180+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:15:40.180180+0000","last_undegraded":"2026-02-17T15:15:40.180180+0000","last_fullsized":"2026-02-17T15:15:40.180180+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":64661,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6870,"num_bytes_recovered":56279177,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":907079680,"data_stored":907104256,"data_compressed":16028,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018886,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325720,"kb_used_data":286396,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046120,"statfs":{"total":96636764160,"available":96303226880,"internally_reserved":0,"allocated":293269504,"data_stored":293095435,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466084,"kb_used_data":286416,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905756,"statfs":{"total":96636764160,"available":96159494144,"internally_reserved":0,"allocated":293289984,"data_stored":293423115,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705694,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75528,"kb_used_data":26360,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94296312,"statfs":{"total":96636764160,"available":96559423488,"internally_reserved":0,"allocated":26992640,"data_stored":27113995,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28135424,"data_stored":28135424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:15:52.602 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 85356, 'reported_epoch': 34, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:15:49.745400+0000', 'last_change': '2026-02-17T15:15:40.196063+0000', 'last_active': '2026-02-17T15:15:49.745400+0000', 'last_peered': '2026-02-17T15:15:49.745400+0000', 'last_clean': '2026-02-17T15:15:34.564274+0000', 'last_became_active': '2026-02-17T15:15:40.195294+0000', 'last_became_peered': '2026-02-17T15:15:40.195294+0000', 'last_unstale': '2026-02-17T15:15:49.745400+0000', 'last_undegraded': '2026-02-17T15:15:49.745400+0000', 'last_fullsized': '2026-02-17T15:15:49.745400+0000', 'mapping_epoch': 33, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 64661, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 6870, 'num_bytes_recovered': 56279177, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:15:52.602 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:15:57.603 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:15:57.604 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:15:57.605 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:15:57.605 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:15:57.696 INFO:teuthology.orchestra.run.trial024.stdout:38654705697 2026-02-17T15:15:57.696 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:15:57.698 INFO:teuthology.orchestra.run.trial024.stdout:38654705696 2026-02-17T15:15:57.698 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:15:57.699 INFO:teuthology.orchestra.run.trial024.stdout:128849018889 2026-02-17T15:15:57.700 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:15:57.701 INFO:teuthology.orchestra.run.trial024.stdout:98784247832 2026-02-17T15:15:57.701 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:15:57.895 INFO:teuthology.orchestra.run.trial024.stdout:98784247831 2026-02-17T15:15:57.896 INFO:teuthology.orchestra.run.trial024.stdout:128849018887 2026-02-17T15:15:57.897 INFO:teuthology.orchestra.run.trial024.stdout:38654705696 2026-02-17T15:15:57.898 INFO:teuthology.orchestra.run.trial024.stdout:38654705695 2026-02-17T15:15:57.909 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018889 got 128849018887 for osd.3 2026-02-17T15:15:57.911 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247832 got 98784247831 for osd.1 2026-02-17T15:15:57.912 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705696 got 38654705695 for osd.2 2026-02-17T15:15:57.913 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705697 got 38654705696 for osd.0 2026-02-17T15:15:58.910 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:15:58.911 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:15:58.913 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:15:58.914 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:15:59.061 INFO:teuthology.orchestra.run.trial024.stdout:98784247831 2026-02-17T15:15:59.073 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247832 got 98784247831 for osd.1 2026-02-17T15:15:59.074 INFO:teuthology.orchestra.run.trial024.stdout:128849018887 2026-02-17T15:15:59.088 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018889 got 128849018887 for osd.3 2026-02-17T15:15:59.109 INFO:teuthology.orchestra.run.trial024.stdout:38654705695 2026-02-17T15:15:59.122 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705696 got 38654705695 for osd.2 2026-02-17T15:15:59.123 INFO:teuthology.orchestra.run.trial024.stdout:38654705696 2026-02-17T15:15:59.137 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705697 got 38654705696 for osd.0 2026-02-17T15:16:00.074 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:00.088 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:00.123 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:00.138 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:00.238 INFO:teuthology.orchestra.run.trial024.stdout:98784247832 2026-02-17T15:16:00.251 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247832 got 98784247832 for osd.1 2026-02-17T15:16:00.252 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:00.273 INFO:teuthology.orchestra.run.trial024.stdout:128849018889 2026-02-17T15:16:00.285 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018889 got 128849018889 for osd.3 2026-02-17T15:16:00.286 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:00.323 INFO:teuthology.orchestra.run.trial024.stdout:38654705697 2026-02-17T15:16:00.330 INFO:teuthology.orchestra.run.trial024.stdout:38654705696 2026-02-17T15:16:00.336 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705697 got 38654705697 for osd.0 2026-02-17T15:16:00.336 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:00.342 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705696 got 38654705696 for osd.2 2026-02-17T15:16:00.343 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:00.343 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:16:00.530 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:16:00.530 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:16:00.541 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2026-02-17T15:15:59.477994+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":58833,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12700,"num_bytes_recovered":104350482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":904480,"kb_used_data":622432,"kb_used_omap":7,"kb_used_meta":281976,"kb_avail":282211040,"statfs":{"total":289910292480,"available":288984104960,"internally_reserved":0,"allocated":637370368,"data_stored":637454881,"data_compressed":15084,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":7634,"internal_metadata":288743982},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-10766,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10766,"num_bytes_recovered":88195209,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001496"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":97012,"reported_epoch":34,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:15:57.680694+0000","last_change":"2026-02-17T15:15:40.196063+0000","last_active":"2026-02-17T15:15:57.680694+0000","last_peered":"2026-02-17T15:15:57.680694+0000","last_clean":"2026-02-17T15:15:34.564274+0000","last_became_active":"2026-02-17T15:15:40.195294+0000","last_became_peered":"2026-02-17T15:15:40.195294+0000","last_unstale":"2026-02-17T15:15:57.680694+0000","last_undegraded":"2026-02-17T15:15:57.680694+0000","last_fullsized":"2026-02-17T15:15:57.680694+0000","mapping_epoch":33,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":58833,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12698,"num_bytes_recovered":104022290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-17T15:15:40.180180+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:15:40.180180+0000","last_peered":"2026-02-17T15:15:40.180180+0000","last_clean":"2026-02-17T15:15:40.180180+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:15:40.180180+0000","last_undegraded":"2026-02-17T15:15:40.180180+0000","last_fullsized":"2026-02-17T15:15:40.180180+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":58833,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12698,"num_bytes_recovered":104022290,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":930951168,"data_stored":930979840,"data_compressed":18158,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247832,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325720,"kb_used_data":286396,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046120,"statfs":{"total":96636764160,"available":96303226880,"internally_reserved":0,"allocated":293269504,"data_stored":293095435,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466084,"kb_used_data":286416,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905756,"statfs":{"total":96636764160,"available":96159494144,"internally_reserved":0,"allocated":293289984,"data_stored":293423115,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705697,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112676,"kb_used_data":49620,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94259164,"statfs":{"total":96636764160,"available":96521383936,"internally_reserved":0,"allocated":50810880,"data_stored":50936331,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52006912,"data_stored":52011008,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:16:00.542 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 97012, 'reported_epoch': 34, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:15:57.680694+0000', 'last_change': '2026-02-17T15:15:40.196063+0000', 'last_active': '2026-02-17T15:15:57.680694+0000', 'last_peered': '2026-02-17T15:15:57.680694+0000', 'last_clean': '2026-02-17T15:15:34.564274+0000', 'last_became_active': '2026-02-17T15:15:40.195294+0000', 'last_became_peered': '2026-02-17T15:15:40.195294+0000', 'last_unstale': '2026-02-17T15:15:57.680694+0000', 'last_undegraded': '2026-02-17T15:15:57.680694+0000', 'last_fullsized': '2026-02-17T15:15:57.680694+0000', 'mapping_epoch': 33, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 58833, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 12698, 'num_bytes_recovered': 104022290, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:16:00.542 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:16:05.543 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:16:05.544 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:16:05.544 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:16:05.545 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:16:05.633 INFO:teuthology.orchestra.run.trial024.stdout:38654705699 2026-02-17T15:16:05.633 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:05.634 INFO:teuthology.orchestra.run.trial024.stdout:38654705698 2026-02-17T15:16:05.635 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:05.636 INFO:teuthology.orchestra.run.trial024.stdout:128849018891 2026-02-17T15:16:05.636 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:05.638 INFO:teuthology.orchestra.run.trial024.stdout:98784247835 2026-02-17T15:16:05.639 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:05.825 INFO:teuthology.orchestra.run.trial024.stdout:38654705698 2026-02-17T15:16:05.838 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705699 got 38654705698 for osd.0 2026-02-17T15:16:05.839 INFO:teuthology.orchestra.run.trial024.stdout:98784247834 2026-02-17T15:16:05.843 INFO:teuthology.orchestra.run.trial024.stdout:128849018890 2026-02-17T15:16:05.853 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247835 got 98784247834 for osd.1 2026-02-17T15:16:05.856 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018891 got 128849018890 for osd.3 2026-02-17T15:16:05.858 INFO:teuthology.orchestra.run.trial024.stdout:38654705697 2026-02-17T15:16:05.870 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705698 got 38654705697 for osd.2 2026-02-17T15:16:06.839 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:06.855 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:06.856 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:06.871 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:06.999 INFO:teuthology.orchestra.run.trial024.stdout:38654705698 2026-02-17T15:16:07.012 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705699 got 38654705698 for osd.0 2026-02-17T15:16:07.037 INFO:teuthology.orchestra.run.trial024.stdout:98784247834 2026-02-17T15:16:07.042 INFO:teuthology.orchestra.run.trial024.stdout:38654705697 2026-02-17T15:16:07.046 INFO:teuthology.orchestra.run.trial024.stdout:128849018890 2026-02-17T15:16:07.049 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247835 got 98784247834 for osd.1 2026-02-17T15:16:07.054 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705698 got 38654705697 for osd.2 2026-02-17T15:16:07.059 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018891 got 128849018890 for osd.3 2026-02-17T15:16:08.013 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:08.050 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:08.055 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:08.060 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:08.168 INFO:teuthology.orchestra.run.trial024.stdout:38654705700 2026-02-17T15:16:08.180 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705699 got 38654705700 for osd.0 2026-02-17T15:16:08.181 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:08.211 INFO:teuthology.orchestra.run.trial024.stdout:38654705699 2026-02-17T15:16:08.214 INFO:teuthology.orchestra.run.trial024.stdout:128849018891 2026-02-17T15:16:08.223 INFO:teuthology.orchestra.run.trial024.stdout:98784247835 2026-02-17T15:16:08.224 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705698 got 38654705699 for osd.2 2026-02-17T15:16:08.224 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:08.227 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018891 got 128849018891 for osd.3 2026-02-17T15:16:08.228 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:08.235 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247835 got 98784247835 for osd.1 2026-02-17T15:16:08.235 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:08.236 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:16:08.422 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:16:08.422 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:16:08.433 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2026-02-17T15:16:07.478941+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":52924,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18609,"num_bytes_recovered":152757010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":971340,"kb_used_data":649036,"kb_used_omap":7,"kb_used_meta":322232,"kb_avail":282144180,"statfs":{"total":289910292480,"available":288915640320,"internally_reserved":0,"allocated":664612864,"data_stored":664697377,"data_compressed":15084,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":7634,"internal_metadata":329966126},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-9711,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9711,"num_bytes_recovered":79552649,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001448"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":108830,"reported_epoch":34,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:16:05.618419+0000","last_change":"2026-02-17T15:15:40.196063+0000","last_active":"2026-02-17T15:16:05.618419+0000","last_peered":"2026-02-17T15:16:05.618419+0000","last_clean":"2026-02-17T15:15:34.564274+0000","last_became_active":"2026-02-17T15:15:40.195294+0000","last_became_peered":"2026-02-17T15:15:40.195294+0000","last_unstale":"2026-02-17T15:16:05.618419+0000","last_undegraded":"2026-02-17T15:16:05.618419+0000","last_fullsized":"2026-02-17T15:16:05.618419+0000","mapping_epoch":33,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":52924,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18607,"num_bytes_recovered":152428818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-17T15:15:40.180180+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:15:40.180180+0000","last_peered":"2026-02-17T15:15:40.180180+0000","last_clean":"2026-02-17T15:15:40.180180+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:15:40.180180+0000","last_undegraded":"2026-02-17T15:15:40.180180+0000","last_fullsized":"2026-02-17T15:15:40.180180+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":52924,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18607,"num_bytes_recovered":152428818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":958050304,"data_stored":958078976,"data_compressed":18158,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247835,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018891,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325720,"kb_used_data":286396,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046120,"statfs":{"total":96636764160,"available":96303226880,"internally_reserved":0,"allocated":293269504,"data_stored":293095435,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466084,"kb_used_data":286416,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905756,"statfs":{"total":96636764160,"available":96159494144,"internally_reserved":0,"allocated":293289984,"data_stored":293423115,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705700,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":179536,"kb_used_data":76224,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":94192304,"statfs":{"total":96636764160,"available":96452919296,"internally_reserved":0,"allocated":78053376,"data_stored":78178827,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":105773510},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":79106048,"data_stored":79110144,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:16:08.434 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 108830, 'reported_epoch': 34, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:16:05.618419+0000', 'last_change': '2026-02-17T15:15:40.196063+0000', 'last_active': '2026-02-17T15:16:05.618419+0000', 'last_peered': '2026-02-17T15:16:05.618419+0000', 'last_clean': '2026-02-17T15:15:34.564274+0000', 'last_became_active': '2026-02-17T15:15:40.195294+0000', 'last_became_peered': '2026-02-17T15:15:40.195294+0000', 'last_unstale': '2026-02-17T15:16:05.618419+0000', 'last_undegraded': '2026-02-17T15:16:05.618419+0000', 'last_fullsized': '2026-02-17T15:16:05.618419+0000', 'mapping_epoch': 33, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 52924, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 18607, 'num_bytes_recovered': 152428818, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:16:08.434 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:16:13.436 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:16:13.437 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:16:13.437 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:16:13.438 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:16:13.525 INFO:teuthology.orchestra.run.trial024.stdout:38654705701 2026-02-17T15:16:13.525 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:13.526 INFO:teuthology.orchestra.run.trial024.stdout:98784247837 2026-02-17T15:16:13.527 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:13.529 INFO:teuthology.orchestra.run.trial024.stdout:38654705702 2026-02-17T15:16:13.530 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:13.533 INFO:teuthology.orchestra.run.trial024.stdout:128849018894 2026-02-17T15:16:13.534 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:13.681 INFO:teuthology.orchestra.run.trial024.stdout:38654705700 2026-02-17T15:16:13.690 INFO:teuthology.orchestra.run.trial024.stdout:98784247836 2026-02-17T15:16:13.694 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705701 got 38654705700 for osd.2 2026-02-17T15:16:13.703 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247837 got 98784247836 for osd.1 2026-02-17T15:16:13.729 INFO:teuthology.orchestra.run.trial024.stdout:128849018893 2026-02-17T15:16:13.731 INFO:teuthology.orchestra.run.trial024.stdout:38654705701 2026-02-17T15:16:13.742 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018894 got 128849018893 for osd.3 2026-02-17T15:16:13.743 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705702 got 38654705701 for osd.0 2026-02-17T15:16:14.695 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:14.704 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:14.742 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:14.744 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:14.841 INFO:teuthology.orchestra.run.trial024.stdout:38654705700 2026-02-17T15:16:14.854 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705701 got 38654705700 for osd.2 2026-02-17T15:16:14.900 INFO:teuthology.orchestra.run.trial024.stdout:98784247836 2026-02-17T15:16:14.913 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247837 got 98784247836 for osd.1 2026-02-17T15:16:14.934 INFO:teuthology.orchestra.run.trial024.stdout:38654705701 2026-02-17T15:16:14.945 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705702 got 38654705701 for osd.0 2026-02-17T15:16:14.947 INFO:teuthology.orchestra.run.trial024.stdout:128849018893 2026-02-17T15:16:14.960 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018894 got 128849018893 for osd.3 2026-02-17T15:16:15.855 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:15.914 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:15.946 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:15.961 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:16.021 INFO:teuthology.orchestra.run.trial024.stdout:38654705701 2026-02-17T15:16:16.034 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705701 got 38654705701 for osd.2 2026-02-17T15:16:16.035 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:16.071 INFO:teuthology.orchestra.run.trial024.stdout:98784247838 2026-02-17T15:16:16.084 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247837 got 98784247838 for osd.1 2026-02-17T15:16:16.085 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:16.111 INFO:teuthology.orchestra.run.trial024.stdout:128849018894 2026-02-17T15:16:16.111 INFO:teuthology.orchestra.run.trial024.stdout:38654705702 2026-02-17T15:16:16.124 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018894 got 128849018894 for osd.3 2026-02-17T15:16:16.124 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:16.125 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705702 got 38654705702 for osd.0 2026-02-17T15:16:16.125 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:16.125 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:16:16.310 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:16:16.310 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:16:16.321 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2026-02-17T15:16:15.479990+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":46922,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24611,"num_bytes_recovered":201925394,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1024924,"kb_used_data":670684,"kb_used_omap":7,"kb_used_meta":354168,"kb_avail":282090596,"statfs":{"total":289910292480,"available":288860770304,"internally_reserved":0,"allocated":686780416,"data_stored":686864929,"data_compressed":15084,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":7634,"internal_metadata":362668590},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8318,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8318,"num_bytes_recovered":68141056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001524"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":120834,"reported_epoch":34,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:16:13.517934+0000","last_change":"2026-02-17T15:15:40.196063+0000","last_active":"2026-02-17T15:16:13.517934+0000","last_peered":"2026-02-17T15:16:13.517934+0000","last_clean":"2026-02-17T15:15:34.564274+0000","last_became_active":"2026-02-17T15:15:40.195294+0000","last_became_peered":"2026-02-17T15:15:40.195294+0000","last_unstale":"2026-02-17T15:16:13.517934+0000","last_undegraded":"2026-02-17T15:16:13.517934+0000","last_fullsized":"2026-02-17T15:16:13.517934+0000","mapping_epoch":33,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":46922,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24609,"num_bytes_recovered":201597202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-17T15:15:40.180180+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:15:40.180180+0000","last_peered":"2026-02-17T15:15:40.180180+0000","last_clean":"2026-02-17T15:15:40.180180+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:15:40.180180+0000","last_undegraded":"2026-02-17T15:15:40.180180+0000","last_fullsized":"2026-02-17T15:15:40.180180+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":46922,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24609,"num_bytes_recovered":201597202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":979722240,"data_stored":979750912,"data_compressed":18158,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018894,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325720,"kb_used_data":286396,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046120,"statfs":{"total":96636764160,"available":96303226880,"internally_reserved":0,"allocated":293269504,"data_stored":293095435,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466084,"kb_used_data":286416,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905756,"statfs":{"total":96636764160,"available":96159494144,"internally_reserved":0,"allocated":293289984,"data_stored":293423115,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705702,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233120,"kb_used_data":97872,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":94138720,"statfs":{"total":96636764160,"available":96398049280,"internally_reserved":0,"allocated":100220928,"data_stored":100346379,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":138475974},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":100777984,"data_stored":100782080,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:16:16.322 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 120834, 'reported_epoch': 34, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:16:13.517934+0000', 'last_change': '2026-02-17T15:15:40.196063+0000', 'last_active': '2026-02-17T15:16:13.517934+0000', 'last_peered': '2026-02-17T15:16:13.517934+0000', 'last_clean': '2026-02-17T15:15:34.564274+0000', 'last_became_active': '2026-02-17T15:15:40.195294+0000', 'last_became_peered': '2026-02-17T15:15:40.195294+0000', 'last_unstale': '2026-02-17T15:16:13.517934+0000', 'last_undegraded': '2026-02-17T15:16:13.517934+0000', 'last_fullsized': '2026-02-17T15:16:13.517934+0000', 'mapping_epoch': 33, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 46922, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 24609, 'num_bytes_recovered': 201597202, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:16:16.322 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:16:21.324 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:16:21.325 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:16:21.325 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:16:21.326 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:16:21.413 INFO:teuthology.orchestra.run.trial024.stdout:98784247840 2026-02-17T15:16:21.414 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:21.415 INFO:teuthology.orchestra.run.trial024.stdout:38654705704 2026-02-17T15:16:21.415 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:21.416 INFO:teuthology.orchestra.run.trial024.stdout:38654705703 2026-02-17T15:16:21.417 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:21.418 INFO:teuthology.orchestra.run.trial024.stdout:128849018896 2026-02-17T15:16:21.418 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:21.612 INFO:teuthology.orchestra.run.trial024.stdout:38654705704 2026-02-17T15:16:21.613 INFO:teuthology.orchestra.run.trial024.stdout:98784247840 2026-02-17T15:16:21.624 INFO:teuthology.orchestra.run.trial024.stdout:38654705703 2026-02-17T15:16:21.625 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705704 got 38654705704 for osd.0 2026-02-17T15:16:21.626 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:21.626 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247840 got 98784247840 for osd.1 2026-02-17T15:16:21.627 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:21.636 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705703 got 38654705703 for osd.2 2026-02-17T15:16:21.637 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:21.670 INFO:teuthology.orchestra.run.trial024.stdout:128849018896 2026-02-17T15:16:21.682 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018896 got 128849018896 for osd.3 2026-02-17T15:16:21.682 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:21.683 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:16:21.869 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:16:21.869 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:16:21.880 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2026-02-17T15:16:21.480750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40994,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30539,"num_bytes_recovered":250487570,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1096796,"kb_used_data":693980,"kb_used_omap":7,"kb_used_meta":402744,"kb_avail":282018724,"statfs":{"total":289910292480,"available":288787173376,"internally_reserved":0,"allocated":710635520,"data_stored":710720033,"data_compressed":15084,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":7634,"internal_metadata":412410414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-10473,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10473,"num_bytes_recovered":85794816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001549"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":132690,"reported_epoch":34,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:16:21.401058+0000","last_change":"2026-02-17T15:15:40.196063+0000","last_active":"2026-02-17T15:16:21.401058+0000","last_peered":"2026-02-17T15:16:21.401058+0000","last_clean":"2026-02-17T15:15:34.564274+0000","last_became_active":"2026-02-17T15:15:40.195294+0000","last_became_peered":"2026-02-17T15:15:40.195294+0000","last_unstale":"2026-02-17T15:16:21.401058+0000","last_undegraded":"2026-02-17T15:16:21.401058+0000","last_fullsized":"2026-02-17T15:16:21.401058+0000","mapping_epoch":33,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40994,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30537,"num_bytes_recovered":250159378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":55,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-17T15:15:40.180180+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:15:40.180180+0000","last_peered":"2026-02-17T15:15:40.180180+0000","last_clean":"2026-02-17T15:15:40.180180+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:15:40.180180+0000","last_undegraded":"2026-02-17T15:15:40.180180+0000","last_fullsized":"2026-02-17T15:15:40.180180+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40994,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30537,"num_bytes_recovered":250159378,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1004011520,"data_stored":1004040192,"data_compressed":18158,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247840,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018896,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325720,"kb_used_data":286396,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046120,"statfs":{"total":96636764160,"available":96303226880,"internally_reserved":0,"allocated":293269504,"data_stored":293095435,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466084,"kb_used_data":286416,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905756,"statfs":{"total":96636764160,"available":96159494144,"internally_reserved":0,"allocated":293289984,"data_stored":293423115,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304992,"kb_used_data":121168,"kb_used_omap":1,"kb_used_meta":183806,"kb_avail":94066848,"statfs":{"total":96636764160,"available":96324452352,"internally_reserved":0,"allocated":124076032,"data_stored":124201483,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":188217798},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":125067264,"data_stored":125071360,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:16:21.881 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 132690, 'reported_epoch': 34, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:16:21.401058+0000', 'last_change': '2026-02-17T15:15:40.196063+0000', 'last_active': '2026-02-17T15:16:21.401058+0000', 'last_peered': '2026-02-17T15:16:21.401058+0000', 'last_clean': '2026-02-17T15:15:34.564274+0000', 'last_became_active': '2026-02-17T15:15:40.195294+0000', 'last_became_peered': '2026-02-17T15:15:40.195294+0000', 'last_unstale': '2026-02-17T15:16:21.401058+0000', 'last_undegraded': '2026-02-17T15:16:21.401058+0000', 'last_fullsized': '2026-02-17T15:16:21.401058+0000', 'mapping_epoch': 33, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 40994, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 30537, 'num_bytes_recovered': 250159378, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:16:21.881 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:16:26.882 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:16:26.883 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:16:26.884 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:16:26.884 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:16:26.973 INFO:teuthology.orchestra.run.trial024.stdout:38654705707 2026-02-17T15:16:26.974 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:26.975 INFO:teuthology.orchestra.run.trial024.stdout:38654705706 2026-02-17T15:16:26.976 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:26.979 INFO:teuthology.orchestra.run.trial024.stdout:128849018898 2026-02-17T15:16:26.980 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:26.982 INFO:teuthology.orchestra.run.trial024.stdout:98784247842 2026-02-17T15:16:26.982 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:27.132 INFO:teuthology.orchestra.run.trial024.stdout:38654705704 2026-02-17T15:16:27.135 INFO:teuthology.orchestra.run.trial024.stdout:38654705705 2026-02-17T15:16:27.146 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705706 got 38654705704 for osd.2 2026-02-17T15:16:27.148 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705707 got 38654705705 for osd.0 2026-02-17T15:16:27.177 INFO:teuthology.orchestra.run.trial024.stdout:128849018897 2026-02-17T15:16:27.189 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018898 got 128849018897 for osd.3 2026-02-17T15:16:27.190 INFO:teuthology.orchestra.run.trial024.stdout:98784247841 2026-02-17T15:16:27.203 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247842 got 98784247841 for osd.1 2026-02-17T15:16:28.147 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:28.149 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:28.190 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:28.204 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:28.302 INFO:teuthology.orchestra.run.trial024.stdout:38654705706 2026-02-17T15:16:28.303 INFO:teuthology.orchestra.run.trial024.stdout:38654705707 2026-02-17T15:16:28.316 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705706 got 38654705706 for osd.2 2026-02-17T15:16:28.317 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:28.317 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705707 got 38654705707 for osd.0 2026-02-17T15:16:28.318 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:28.341 INFO:teuthology.orchestra.run.trial024.stdout:128849018898 2026-02-17T15:16:28.356 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018898 got 128849018898 for osd.3 2026-02-17T15:16:28.357 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:28.394 INFO:teuthology.orchestra.run.trial024.stdout:98784247842 2026-02-17T15:16:28.406 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247842 got 98784247842 for osd.1 2026-02-17T15:16:28.406 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:28.406 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:16:28.589 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:16:28.589 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:16:28.599 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2026-02-17T15:16:27.481549+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36811,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34722,"num_bytes_recovered":284754706,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1111856,"kb_used_data":709040,"kb_used_omap":7,"kb_used_meta":402744,"kb_avail":282003664,"statfs":{"total":289910292480,"available":288771751936,"internally_reserved":0,"allocated":726056960,"data_stored":726141473,"data_compressed":15084,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":7634,"internal_metadata":412410414},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-10111,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10111,"num_bytes_recovered":82829312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001558"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":141056,"reported_epoch":34,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:16:26.965167+0000","last_change":"2026-02-17T15:15:40.196063+0000","last_active":"2026-02-17T15:16:26.965167+0000","last_peered":"2026-02-17T15:16:26.965167+0000","last_clean":"2026-02-17T15:15:34.564274+0000","last_became_active":"2026-02-17T15:15:40.195294+0000","last_became_peered":"2026-02-17T15:15:40.195294+0000","last_unstale":"2026-02-17T15:16:26.965167+0000","last_undegraded":"2026-02-17T15:16:26.965167+0000","last_fullsized":"2026-02-17T15:16:26.965167+0000","mapping_epoch":33,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36811,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34720,"num_bytes_recovered":284426514,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-17T15:15:40.180214+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:15:40.180214+0000","last_peered":"2026-02-17T15:15:40.180214+0000","last_clean":"2026-02-17T15:15:40.180214+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:15:40.180214+0000","last_undegraded":"2026-02-17T15:15:40.180214+0000","last_fullsized":"2026-02-17T15:15:40.180214+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36811,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34720,"num_bytes_recovered":284426514,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1021132800,"data_stored":1021161472,"data_compressed":18158,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247842,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018898,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325720,"kb_used_data":286396,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046120,"statfs":{"total":96636764160,"available":96303226880,"internally_reserved":0,"allocated":293269504,"data_stored":293095435,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466084,"kb_used_data":286416,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905756,"statfs":{"total":96636764160,"available":96159494144,"internally_reserved":0,"allocated":293289984,"data_stored":293423115,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705707,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":320052,"kb_used_data":136228,"kb_used_omap":1,"kb_used_meta":183806,"kb_avail":94051788,"statfs":{"total":96636764160,"available":96309030912,"internally_reserved":0,"allocated":139497472,"data_stored":139622923,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":188217798},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":142188544,"data_stored":142192640,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:16:28.601 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 141056, 'reported_epoch': 34, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:16:26.965167+0000', 'last_change': '2026-02-17T15:15:40.196063+0000', 'last_active': '2026-02-17T15:16:26.965167+0000', 'last_peered': '2026-02-17T15:16:26.965167+0000', 'last_clean': '2026-02-17T15:15:34.564274+0000', 'last_became_active': '2026-02-17T15:15:40.195294+0000', 'last_became_peered': '2026-02-17T15:15:40.195294+0000', 'last_unstale': '2026-02-17T15:16:26.965167+0000', 'last_undegraded': '2026-02-17T15:16:26.965167+0000', 'last_fullsized': '2026-02-17T15:16:26.965167+0000', 'mapping_epoch': 33, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 36811, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 34720, 'num_bytes_recovered': 284426514, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:16:28.601 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:16:33.602 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:16:33.603 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:16:33.603 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:16:33.604 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:16:33.693 INFO:teuthology.orchestra.run.trial024.stdout:38654705708 2026-02-17T15:16:33.693 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:33.695 INFO:teuthology.orchestra.run.trial024.stdout:128849018901 2026-02-17T15:16:33.695 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:33.696 INFO:teuthology.orchestra.run.trial024.stdout:38654705709 2026-02-17T15:16:33.697 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:33.698 INFO:teuthology.orchestra.run.trial024.stdout:98784247844 2026-02-17T15:16:33.699 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:33.848 INFO:teuthology.orchestra.run.trial024.stdout:38654705707 2026-02-17T15:16:33.861 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705708 got 38654705707 for osd.2 2026-02-17T15:16:33.895 INFO:teuthology.orchestra.run.trial024.stdout:98784247843 2026-02-17T15:16:33.899 INFO:teuthology.orchestra.run.trial024.stdout:128849018900 2026-02-17T15:16:33.899 INFO:teuthology.orchestra.run.trial024.stdout:38654705708 2026-02-17T15:16:33.908 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247844 got 98784247843 for osd.1 2026-02-17T15:16:33.914 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018901 got 128849018900 for osd.3 2026-02-17T15:16:33.915 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705709 got 38654705708 for osd.0 2026-02-17T15:16:34.862 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:34.909 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:34.915 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:34.916 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:35.010 INFO:teuthology.orchestra.run.trial024.stdout:38654705707 2026-02-17T15:16:35.024 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705708 got 38654705707 for osd.2 2026-02-17T15:16:35.058 INFO:teuthology.orchestra.run.trial024.stdout:98784247843 2026-02-17T15:16:35.071 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247844 got 98784247843 for osd.1 2026-02-17T15:16:35.112 INFO:teuthology.orchestra.run.trial024.stdout:38654705708 2026-02-17T15:16:35.117 INFO:teuthology.orchestra.run.trial024.stdout:128849018900 2026-02-17T15:16:35.123 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705709 got 38654705708 for osd.0 2026-02-17T15:16:35.129 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018901 got 128849018900 for osd.3 2026-02-17T15:16:36.025 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:36.072 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:36.125 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:36.130 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:36.188 INFO:teuthology.orchestra.run.trial024.stdout:38654705708 2026-02-17T15:16:36.203 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705708 got 38654705708 for osd.2 2026-02-17T15:16:36.203 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:36.226 INFO:teuthology.orchestra.run.trial024.stdout:98784247845 2026-02-17T15:16:36.239 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247844 got 98784247845 for osd.1 2026-02-17T15:16:36.239 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:36.271 INFO:teuthology.orchestra.run.trial024.stdout:38654705709 2026-02-17T15:16:36.284 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705709 got 38654705709 for osd.0 2026-02-17T15:16:36.284 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:36.335 INFO:teuthology.orchestra.run.trial024.stdout:128849018901 2026-02-17T15:16:36.346 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018901 got 128849018901 for osd.3 2026-02-17T15:16:36.347 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:36.347 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:16:36.493 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:16:36.493 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:16:36.504 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2026-02-17T15:16:35.482597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":31884,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39649,"num_bytes_recovered":325116690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1160516,"kb_used_data":729924,"kb_used_omap":7,"kb_used_meta":430520,"kb_avail":281955004,"statfs":{"total":289910292480,"available":288721924096,"internally_reserved":0,"allocated":747442176,"data_stored":747526689,"data_compressed":15084,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":7634,"internal_metadata":440853038},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8223,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8223,"num_bytes_recovered":67362816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001539"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":150910,"reported_epoch":34,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:16:33.679660+0000","last_change":"2026-02-17T15:15:40.196063+0000","last_active":"2026-02-17T15:16:33.679660+0000","last_peered":"2026-02-17T15:16:33.679660+0000","last_clean":"2026-02-17T15:15:34.564274+0000","last_became_active":"2026-02-17T15:15:40.195294+0000","last_became_peered":"2026-02-17T15:15:40.195294+0000","last_unstale":"2026-02-17T15:16:33.679660+0000","last_undegraded":"2026-02-17T15:16:33.679660+0000","last_fullsized":"2026-02-17T15:16:33.679660+0000","mapping_epoch":33,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":31884,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39647,"num_bytes_recovered":324788498,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-17T15:15:40.180214+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:15:40.180214+0000","last_peered":"2026-02-17T15:15:40.180214+0000","last_clean":"2026-02-17T15:15:40.180214+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:15:40.180214+0000","last_undegraded":"2026-02-17T15:15:40.180214+0000","last_fullsized":"2026-02-17T15:15:40.180214+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":31884,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39647,"num_bytes_recovered":324788498,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1041338368,"data_stored":1041367040,"data_compressed":18158,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018901,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325720,"kb_used_data":286396,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046120,"statfs":{"total":96636764160,"available":96303226880,"internally_reserved":0,"allocated":293269504,"data_stored":293095435,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466084,"kb_used_data":286416,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905756,"statfs":{"total":96636764160,"available":96159494144,"internally_reserved":0,"allocated":293289984,"data_stored":293423115,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705709,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":368712,"kb_used_data":157112,"kb_used_omap":1,"kb_used_meta":211582,"kb_avail":94003128,"statfs":{"total":96636764160,"available":96259203072,"internally_reserved":0,"allocated":160882688,"data_stored":161008139,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":216660422},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":162394112,"data_stored":162398208,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:16:36.505 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 150910, 'reported_epoch': 34, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:16:33.679660+0000', 'last_change': '2026-02-17T15:15:40.196063+0000', 'last_active': '2026-02-17T15:16:33.679660+0000', 'last_peered': '2026-02-17T15:16:33.679660+0000', 'last_clean': '2026-02-17T15:15:34.564274+0000', 'last_became_active': '2026-02-17T15:15:40.195294+0000', 'last_became_peered': '2026-02-17T15:15:40.195294+0000', 'last_unstale': '2026-02-17T15:16:33.679660+0000', 'last_undegraded': '2026-02-17T15:16:33.679660+0000', 'last_fullsized': '2026-02-17T15:16:33.679660+0000', 'mapping_epoch': 33, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 31884, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 39647, 'num_bytes_recovered': 324788498, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:16:36.505 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:16:41.506 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:16:41.507 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:16:41.507 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:16:41.508 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:16:41.596 INFO:teuthology.orchestra.run.trial024.stdout:38654705712 2026-02-17T15:16:41.597 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:41.598 INFO:teuthology.orchestra.run.trial024.stdout:128849018903 2026-02-17T15:16:41.599 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:41.600 INFO:teuthology.orchestra.run.trial024.stdout:98784247847 2026-02-17T15:16:41.601 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:41.604 INFO:teuthology.orchestra.run.trial024.stdout:38654705711 2026-02-17T15:16:41.605 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:41.754 INFO:teuthology.orchestra.run.trial024.stdout:128849018902 2026-02-17T15:16:41.760 INFO:teuthology.orchestra.run.trial024.stdout:38654705710 2026-02-17T15:16:41.768 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018903 got 128849018902 for osd.3 2026-02-17T15:16:41.773 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705712 got 38654705710 for osd.0 2026-02-17T15:16:41.799 INFO:teuthology.orchestra.run.trial024.stdout:38654705709 2026-02-17T15:16:41.811 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705711 got 38654705709 for osd.2 2026-02-17T15:16:41.813 INFO:teuthology.orchestra.run.trial024.stdout:98784247846 2026-02-17T15:16:41.826 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247847 got 98784247846 for osd.1 2026-02-17T15:16:42.769 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:42.774 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:42.811 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:42.827 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:42.918 INFO:teuthology.orchestra.run.trial024.stdout:128849018902 2026-02-17T15:16:42.931 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018903 got 128849018902 for osd.3 2026-02-17T15:16:42.980 INFO:teuthology.orchestra.run.trial024.stdout:38654705709 2026-02-17T15:16:42.981 INFO:teuthology.orchestra.run.trial024.stdout:38654705710 2026-02-17T15:16:42.993 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705711 got 38654705709 for osd.2 2026-02-17T15:16:42.994 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705712 got 38654705710 for osd.0 2026-02-17T15:16:43.020 INFO:teuthology.orchestra.run.trial024.stdout:98784247846 2026-02-17T15:16:43.033 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247847 got 98784247846 for osd.1 2026-02-17T15:16:43.932 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:43.994 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:43.995 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:44.033 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:44.085 INFO:teuthology.orchestra.run.trial024.stdout:128849018904 2026-02-17T15:16:44.097 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018903 got 128849018904 for osd.3 2026-02-17T15:16:44.097 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:44.149 INFO:teuthology.orchestra.run.trial024.stdout:38654705711 2026-02-17T15:16:44.154 INFO:teuthology.orchestra.run.trial024.stdout:38654705712 2026-02-17T15:16:44.161 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705711 got 38654705711 for osd.2 2026-02-17T15:16:44.162 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:44.166 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705712 got 38654705712 for osd.0 2026-02-17T15:16:44.167 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:44.190 INFO:teuthology.orchestra.run.trial024.stdout:98784247847 2026-02-17T15:16:44.202 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247847 got 98784247847 for osd.1 2026-02-17T15:16:44.202 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:44.203 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:16:44.390 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:16:44.391 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:16:44.400 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2026-02-17T15:16:43.483565+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":25178,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46355,"num_bytes_recovered":380052242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1220644,"kb_used_data":753956,"kb_used_omap":7,"kb_used_meta":466616,"kb_avail":281894876,"statfs":{"total":289910292480,"available":288660353024,"internally_reserved":0,"allocated":772050944,"data_stored":772135457,"data_compressed":15084,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":7634,"internal_metadata":477815342},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-11179,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11179,"num_bytes_recovered":91578368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001501"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":164322,"reported_epoch":34,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:16:42.565695+0000","last_change":"2026-02-17T15:15:40.196063+0000","last_active":"2026-02-17T15:16:42.565695+0000","last_peered":"2026-02-17T15:16:42.565695+0000","last_clean":"2026-02-17T15:15:34.564274+0000","last_became_active":"2026-02-17T15:15:40.195294+0000","last_became_peered":"2026-02-17T15:15:40.195294+0000","last_unstale":"2026-02-17T15:16:42.565695+0000","last_undegraded":"2026-02-17T15:16:42.565695+0000","last_fullsized":"2026-02-17T15:16:42.565695+0000","mapping_epoch":33,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":25178,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46353,"num_bytes_recovered":379724050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-17T15:15:40.180214+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:15:40.180214+0000","last_peered":"2026-02-17T15:15:40.180214+0000","last_clean":"2026-02-17T15:15:40.180214+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:15:40.180214+0000","last_undegraded":"2026-02-17T15:15:40.180214+0000","last_fullsized":"2026-02-17T15:15:40.180214+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":25178,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46353,"num_bytes_recovered":379724050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1066065920,"data_stored":1066094592,"data_compressed":18158,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018904,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325720,"kb_used_data":286396,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046120,"statfs":{"total":96636764160,"available":96303226880,"internally_reserved":0,"allocated":293269504,"data_stored":293095435,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466084,"kb_used_data":286416,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905756,"statfs":{"total":96636764160,"available":96159494144,"internally_reserved":0,"allocated":293289984,"data_stored":293423115,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705712,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":428840,"kb_used_data":181144,"kb_used_omap":1,"kb_used_meta":247678,"kb_avail":93943000,"statfs":{"total":96636764160,"available":96197632000,"internally_reserved":0,"allocated":185491456,"data_stored":185616907,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":1594,"internal_metadata":253622726},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":187121664,"data_stored":187125760,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:16:44.402 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 164322, 'reported_epoch': 34, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:16:42.565695+0000', 'last_change': '2026-02-17T15:15:40.196063+0000', 'last_active': '2026-02-17T15:16:42.565695+0000', 'last_peered': '2026-02-17T15:16:42.565695+0000', 'last_clean': '2026-02-17T15:15:34.564274+0000', 'last_became_active': '2026-02-17T15:15:40.195294+0000', 'last_became_peered': '2026-02-17T15:15:40.195294+0000', 'last_unstale': '2026-02-17T15:16:42.565695+0000', 'last_undegraded': '2026-02-17T15:16:42.565695+0000', 'last_fullsized': '2026-02-17T15:16:42.565695+0000', 'mapping_epoch': 33, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 25178, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 46353, 'num_bytes_recovered': 379724050, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:16:44.402 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:16:45.193 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:16:45.188+0000 7f2493536640 -1 osd.3 pg_epoch: 34 pg[2.0s0( v 26'71529 (26'71456,26'71529] local-lis/les=33/34 n=71529 ec=16/16 lis/c=33/30 les/c/f=34/31/0 sis=33) [3,0,2]/[3,1,2]p3(0) backfill=[0(1)] r=0 lpr=33 pi=[30,33)/1 crt=26'71529 lcod 0'0 mlcod 0'0 active+remapped+backfilling rops=1 mbc={0={},1={},2={}}] continue_recovery_op: 2:ad551702:::test:head has inconsistent hinfo 2026-02-17T15:16:45.193 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:16:45.188+0000 7f2493536640 -1 log_channel(cluster) log [ERR] : 2.0s0 missing primary copy of 2:ad551702:::test:head, will try copies on 1(1),2(2) 2026-02-17T15:16:49.403 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:16:49.404 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:16:49.404 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:16:49.405 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:16:49.492 INFO:teuthology.orchestra.run.trial024.stdout:128849018906 2026-02-17T15:16:49.493 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:49.494 INFO:teuthology.orchestra.run.trial024.stdout:38654705714 2026-02-17T15:16:49.495 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:49.496 INFO:teuthology.orchestra.run.trial024.stdout:38654705713 2026-02-17T15:16:49.496 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:49.498 INFO:teuthology.orchestra.run.trial024.stdout:98784247849 2026-02-17T15:16:49.499 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:49.645 INFO:teuthology.orchestra.run.trial024.stdout:128849018906 2026-02-17T15:16:49.658 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018906 got 128849018906 for osd.3 2026-02-17T15:16:49.658 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:49.689 INFO:teuthology.orchestra.run.trial024.stdout:38654705714 2026-02-17T15:16:49.696 INFO:teuthology.orchestra.run.trial024.stdout:38654705713 2026-02-17T15:16:49.698 INFO:teuthology.orchestra.run.trial024.stdout:98784247849 2026-02-17T15:16:49.701 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705714 got 38654705714 for osd.0 2026-02-17T15:16:49.701 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:49.710 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705713 got 38654705713 for osd.2 2026-02-17T15:16:49.710 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:49.711 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247849 got 98784247849 for osd.1 2026-02-17T15:16:49.711 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:49.711 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:16:49.906 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:16:49.906 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:16:49.916 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2026-02-17T15:16:49.484264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19949,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51584,"num_bytes_recovered":422880179,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1280640,"kb_used_data":777868,"kb_used_omap":7,"kb_used_meta":502712,"kb_avail":281834880,"statfs":{"total":289910292480,"available":288598917120,"internally_reserved":0,"allocated":796536832,"data_stored":796625441,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7634,"internal_metadata":514777646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-11935,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11935,"num_bytes_recovered":97763489,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001430"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":174781,"reported_epoch":34,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:16:49.479464+0000","last_change":"2026-02-17T15:15:40.196063+0000","last_active":"2026-02-17T15:16:49.479464+0000","last_peered":"2026-02-17T15:16:49.479464+0000","last_clean":"2026-02-17T15:15:34.564274+0000","last_became_active":"2026-02-17T15:15:40.195294+0000","last_became_peered":"2026-02-17T15:15:40.195294+0000","last_unstale":"2026-02-17T15:16:49.479464+0000","last_undegraded":"2026-02-17T15:16:49.479464+0000","last_fullsized":"2026-02-17T15:16:49.479464+0000","mapping_epoch":33,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19949,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51582,"num_bytes_recovered":422551987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-17T15:15:40.180214+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:15:40.180214+0000","last_peered":"2026-02-17T15:15:40.180214+0000","last_clean":"2026-02-17T15:15:40.180214+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:15:40.180214+0000","last_undegraded":"2026-02-17T15:15:40.180214+0000","last_fullsized":"2026-02-17T15:15:40.180214+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19949,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51582,"num_bytes_recovered":422551987,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1090220032,"data_stored":1090252800,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247849,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018906,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325708,"kb_used_data":286396,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046132,"statfs":{"total":96636764160,"available":96303239168,"internally_reserved":0,"allocated":293269504,"data_stored":293095435,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466084,"kb_used_data":286416,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905756,"statfs":{"total":96636764160,"available":96159494144,"internally_reserved":0,"allocated":293289984,"data_stored":293423115,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":488848,"kb_used_data":205056,"kb_used_omap":1,"kb_used_meta":283774,"kb_avail":93882992,"statfs":{"total":96636764160,"available":96136183808,"internally_reserved":0,"allocated":209977344,"data_stored":210106891,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":290585030},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":211275776,"data_stored":211283968,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:16:49.917 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 174781, 'reported_epoch': 34, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:16:49.479464+0000', 'last_change': '2026-02-17T15:15:40.196063+0000', 'last_active': '2026-02-17T15:16:49.479464+0000', 'last_peered': '2026-02-17T15:16:49.479464+0000', 'last_clean': '2026-02-17T15:15:34.564274+0000', 'last_became_active': '2026-02-17T15:15:40.195294+0000', 'last_became_peered': '2026-02-17T15:15:40.195294+0000', 'last_unstale': '2026-02-17T15:16:49.479464+0000', 'last_undegraded': '2026-02-17T15:16:49.479464+0000', 'last_fullsized': '2026-02-17T15:16:49.479464+0000', 'mapping_epoch': 33, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 19949, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 51582, 'num_bytes_recovered': 422551987, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:16:49.918 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:16:54.919 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:16:54.919 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:16:54.920 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:16:54.920 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:16:55.008 INFO:teuthology.orchestra.run.trial024.stdout:98784247851 2026-02-17T15:16:55.009 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:55.010 INFO:teuthology.orchestra.run.trial024.stdout:38654705715 2026-02-17T15:16:55.011 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:55.012 INFO:teuthology.orchestra.run.trial024.stdout:128849018908 2026-02-17T15:16:55.013 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:55.013 INFO:teuthology.orchestra.run.trial024.stdout:38654705716 2026-02-17T15:16:55.014 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:55.208 INFO:teuthology.orchestra.run.trial024.stdout:98784247850 2026-02-17T15:16:55.211 INFO:teuthology.orchestra.run.trial024.stdout:38654705714 2026-02-17T15:16:55.213 INFO:teuthology.orchestra.run.trial024.stdout:38654705715 2026-02-17T15:16:55.217 INFO:teuthology.orchestra.run.trial024.stdout:128849018907 2026-02-17T15:16:55.221 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247851 got 98784247850 for osd.1 2026-02-17T15:16:55.224 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705715 got 38654705714 for osd.2 2026-02-17T15:16:55.226 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705716 got 38654705715 for osd.0 2026-02-17T15:16:55.230 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018908 got 128849018907 for osd.3 2026-02-17T15:16:56.222 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:16:56.225 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:16:56.226 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:16:56.231 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:16:56.396 INFO:teuthology.orchestra.run.trial024.stdout:98784247852 2026-02-17T15:16:56.410 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247851 got 98784247852 for osd.1 2026-02-17T15:16:56.411 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:56.421 INFO:teuthology.orchestra.run.trial024.stdout:38654705716 2026-02-17T15:16:56.424 INFO:teuthology.orchestra.run.trial024.stdout:38654705715 2026-02-17T15:16:56.433 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705716 got 38654705716 for osd.0 2026-02-17T15:16:56.434 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:56.437 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705715 got 38654705715 for osd.2 2026-02-17T15:16:56.437 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:56.443 INFO:teuthology.orchestra.run.trial024.stdout:128849018908 2026-02-17T15:16:56.456 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018908 got 128849018908 for osd.3 2026-02-17T15:16:56.456 DEBUG:teuthology.parallel:result is None 2026-02-17T15:16:56.457 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:16:56.652 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:16:56.652 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:16:56.662 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2026-02-17T15:16:55.484980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15839,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55694,"num_bytes_recovered":456549299,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1298180,"kb_used_data":795408,"kb_used_omap":7,"kb_used_meta":502712,"kb_avail":281817340,"statfs":{"total":289910292480,"available":288580956160,"internally_reserved":0,"allocated":814497792,"data_stored":814586401,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7634,"internal_metadata":514777646},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-9339,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9339,"num_bytes_recovered":76497057,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001415"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":183001,"reported_epoch":34,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:16:54.997210+0000","last_change":"2026-02-17T15:15:40.196063+0000","last_active":"2026-02-17T15:16:54.997210+0000","last_peered":"2026-02-17T15:16:54.997210+0000","last_clean":"2026-02-17T15:15:34.564274+0000","last_became_active":"2026-02-17T15:15:40.195294+0000","last_became_peered":"2026-02-17T15:15:40.195294+0000","last_unstale":"2026-02-17T15:16:54.997210+0000","last_undegraded":"2026-02-17T15:16:54.997210+0000","last_fullsized":"2026-02-17T15:16:54.997210+0000","mapping_epoch":33,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15839,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55692,"num_bytes_recovered":456221107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-17T15:15:40.180214+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:15:40.180214+0000","last_peered":"2026-02-17T15:15:40.180214+0000","last_clean":"2026-02-17T15:15:40.180214+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:15:40.180214+0000","last_undegraded":"2026-02-17T15:15:40.180214+0000","last_fullsized":"2026-02-17T15:15:40.180214+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15839,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55692,"num_bytes_recovered":456221107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1107050496,"data_stored":1107083264,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018908,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325708,"kb_used_data":286396,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046132,"statfs":{"total":96636764160,"available":96303239168,"internally_reserved":0,"allocated":293269504,"data_stored":293095435,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466084,"kb_used_data":286416,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905756,"statfs":{"total":96636764160,"available":96159494144,"internally_reserved":0,"allocated":293289984,"data_stored":293423115,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705716,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":506388,"kb_used_data":222596,"kb_used_omap":1,"kb_used_meta":283774,"kb_avail":93865452,"statfs":{"total":96636764160,"available":96118222848,"internally_reserved":0,"allocated":227938304,"data_stored":228067851,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":290585030},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":228106240,"data_stored":228114432,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:16:56.663 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 183001, 'reported_epoch': 34, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:16:54.997210+0000', 'last_change': '2026-02-17T15:15:40.196063+0000', 'last_active': '2026-02-17T15:16:54.997210+0000', 'last_peered': '2026-02-17T15:16:54.997210+0000', 'last_clean': '2026-02-17T15:15:34.564274+0000', 'last_became_active': '2026-02-17T15:15:40.195294+0000', 'last_became_peered': '2026-02-17T15:15:40.195294+0000', 'last_unstale': '2026-02-17T15:16:54.997210+0000', 'last_undegraded': '2026-02-17T15:16:54.997210+0000', 'last_fullsized': '2026-02-17T15:16:54.997210+0000', 'mapping_epoch': 33, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 15839, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 55692, 'num_bytes_recovered': 456221107, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:16:56.664 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:17:01.664 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:17:01.665 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:17:01.666 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:17:01.666 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:17:01.752 INFO:teuthology.orchestra.run.trial024.stdout:38654705718 2026-02-17T15:17:01.753 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:01.754 INFO:teuthology.orchestra.run.trial024.stdout:38654705719 2026-02-17T15:17:01.755 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:01.756 INFO:teuthology.orchestra.run.trial024.stdout:98784247854 2026-02-17T15:17:01.757 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:01.765 INFO:teuthology.orchestra.run.trial024.stdout:128849018910 2026-02-17T15:17:01.765 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:01.919 INFO:teuthology.orchestra.run.trial024.stdout:38654705717 2026-02-17T15:17:01.921 INFO:teuthology.orchestra.run.trial024.stdout:38654705716 2026-02-17T15:17:01.932 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705719 got 38654705717 for osd.0 2026-02-17T15:17:01.935 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705718 got 38654705716 for osd.2 2026-02-17T15:17:01.960 INFO:teuthology.orchestra.run.trial024.stdout:128849018909 2026-02-17T15:17:01.964 INFO:teuthology.orchestra.run.trial024.stdout:98784247853 2026-02-17T15:17:01.973 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018910 got 128849018909 for osd.3 2026-02-17T15:17:01.978 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247854 got 98784247853 for osd.1 2026-02-17T15:17:02.933 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:02.936 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:02.974 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:02.978 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:03.088 INFO:teuthology.orchestra.run.trial024.stdout:38654705717 2026-02-17T15:17:03.101 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705719 got 38654705717 for osd.0 2026-02-17T15:17:03.127 INFO:teuthology.orchestra.run.trial024.stdout:128849018909 2026-02-17T15:17:03.128 INFO:teuthology.orchestra.run.trial024.stdout:38654705716 2026-02-17T15:17:03.139 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018910 got 128849018909 for osd.3 2026-02-17T15:17:03.140 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705718 got 38654705716 for osd.2 2026-02-17T15:17:03.148 INFO:teuthology.orchestra.run.trial024.stdout:98784247853 2026-02-17T15:17:03.161 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247854 got 98784247853 for osd.1 2026-02-17T15:17:04.102 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:04.140 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:04.141 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:04.162 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:04.253 INFO:teuthology.orchestra.run.trial024.stdout:38654705719 2026-02-17T15:17:04.265 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705719 got 38654705719 for osd.0 2026-02-17T15:17:04.266 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:04.302 INFO:teuthology.orchestra.run.trial024.stdout:38654705718 2026-02-17T15:17:04.303 INFO:teuthology.orchestra.run.trial024.stdout:128849018911 2026-02-17T15:17:04.315 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705718 got 38654705718 for osd.2 2026-02-17T15:17:04.316 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:04.317 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018910 got 128849018911 for osd.3 2026-02-17T15:17:04.317 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:04.346 INFO:teuthology.orchestra.run.trial024.stdout:98784247854 2026-02-17T15:17:04.359 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247854 got 98784247854 for osd.1 2026-02-17T15:17:04.359 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:04.359 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:17:04.548 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:17:04.549 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:17:04.559 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2026-02-17T15:17:03.485957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10294,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61239,"num_bytes_recovered":501973939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1355896,"kb_used_data":812868,"kb_used_omap":7,"kb_used_meta":542968,"kb_avail":281759624,"statfs":{"total":289910292480,"available":288521854976,"internally_reserved":0,"allocated":832376832,"data_stored":832465441,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7634,"internal_metadata":555999790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-9655,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9655,"num_bytes_recovered":79093760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001476"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":194091,"reported_epoch":34,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:17:02.567003+0000","last_change":"2026-02-17T15:15:40.196063+0000","last_active":"2026-02-17T15:17:02.567003+0000","last_peered":"2026-02-17T15:17:02.567003+0000","last_clean":"2026-02-17T15:15:34.564274+0000","last_became_active":"2026-02-17T15:15:40.195294+0000","last_became_peered":"2026-02-17T15:15:40.195294+0000","last_unstale":"2026-02-17T15:17:02.567003+0000","last_undegraded":"2026-02-17T15:17:02.567003+0000","last_fullsized":"2026-02-17T15:17:02.567003+0000","mapping_epoch":33,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10294,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61237,"num_bytes_recovered":501645747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-17T15:15:40.180214+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:15:40.180214+0000","last_peered":"2026-02-17T15:15:40.180214+0000","last_clean":"2026-02-17T15:15:40.180214+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:15:40.180214+0000","last_undegraded":"2026-02-17T15:15:40.180214+0000","last_fullsized":"2026-02-17T15:15:40.180214+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10294,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":61237,"num_bytes_recovered":501645747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1127120896,"data_stored":1127153664,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247854,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018911,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325708,"kb_used_data":286396,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046132,"statfs":{"total":96636764160,"available":96303239168,"internally_reserved":0,"allocated":293269504,"data_stored":293095435,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466084,"kb_used_data":286416,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905756,"statfs":{"total":96636764160,"available":96159494144,"internally_reserved":0,"allocated":293289984,"data_stored":293423115,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705719,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564104,"kb_used_data":240056,"kb_used_omap":1,"kb_used_meta":324030,"kb_avail":93807736,"statfs":{"total":96636764160,"available":96059121664,"internally_reserved":0,"allocated":245817344,"data_stored":245946891,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":331807174},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":248176640,"data_stored":248184832,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:17:04.560 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 194091, 'reported_epoch': 34, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:17:02.567003+0000', 'last_change': '2026-02-17T15:15:40.196063+0000', 'last_active': '2026-02-17T15:17:02.567003+0000', 'last_peered': '2026-02-17T15:17:02.567003+0000', 'last_clean': '2026-02-17T15:15:34.564274+0000', 'last_became_active': '2026-02-17T15:15:40.195294+0000', 'last_became_peered': '2026-02-17T15:15:40.195294+0000', 'last_unstale': '2026-02-17T15:17:02.567003+0000', 'last_undegraded': '2026-02-17T15:17:02.567003+0000', 'last_fullsized': '2026-02-17T15:17:02.567003+0000', 'mapping_epoch': 33, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 10294, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 61237, 'num_bytes_recovered': 501645747, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:17:04.561 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:17:09.562 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:17:09.563 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:17:09.563 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:17:09.564 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:17:09.648 INFO:teuthology.orchestra.run.trial024.stdout:38654705720 2026-02-17T15:17:09.648 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:09.649 INFO:teuthology.orchestra.run.trial024.stdout:38654705721 2026-02-17T15:17:09.650 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:09.655 INFO:teuthology.orchestra.run.trial024.stdout:128849018913 2026-02-17T15:17:09.656 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:09.657 INFO:teuthology.orchestra.run.trial024.stdout:98784247856 2026-02-17T15:17:09.658 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:09.807 INFO:teuthology.orchestra.run.trial024.stdout:38654705719 2026-02-17T15:17:09.808 INFO:teuthology.orchestra.run.trial024.stdout:38654705720 2026-02-17T15:17:09.820 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705720 got 38654705719 for osd.2 2026-02-17T15:17:09.821 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705721 got 38654705720 for osd.0 2026-02-17T15:17:09.851 INFO:teuthology.orchestra.run.trial024.stdout:98784247855 2026-02-17T15:17:09.862 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247856 got 98784247855 for osd.1 2026-02-17T15:17:09.868 INFO:teuthology.orchestra.run.trial024.stdout:128849018912 2026-02-17T15:17:09.883 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018913 got 128849018912 for osd.3 2026-02-17T15:17:10.820 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:10.821 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:10.863 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:10.884 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:10.973 INFO:teuthology.orchestra.run.trial024.stdout:38654705719 2026-02-17T15:17:10.985 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705720 got 38654705719 for osd.2 2026-02-17T15:17:10.986 INFO:teuthology.orchestra.run.trial024.stdout:38654705720 2026-02-17T15:17:10.999 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705721 got 38654705720 for osd.0 2026-02-17T15:17:11.018 INFO:teuthology.orchestra.run.trial024.stdout:98784247855 2026-02-17T15:17:11.031 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247856 got 98784247855 for osd.1 2026-02-17T15:17:11.069 INFO:teuthology.orchestra.run.trial024.stdout:128849018912 2026-02-17T15:17:11.082 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018913 got 128849018912 for osd.3 2026-02-17T15:17:11.987 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:12.000 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:12.031 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:12.082 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:12.143 INFO:teuthology.orchestra.run.trial024.stdout:38654705720 2026-02-17T15:17:12.155 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705720 got 38654705720 for osd.2 2026-02-17T15:17:12.156 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:12.188 INFO:teuthology.orchestra.run.trial024.stdout:98784247857 2026-02-17T15:17:12.192 INFO:teuthology.orchestra.run.trial024.stdout:38654705721 2026-02-17T15:17:12.201 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247856 got 98784247857 for osd.1 2026-02-17T15:17:12.201 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:12.205 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705721 got 38654705721 for osd.0 2026-02-17T15:17:12.205 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:12.243 INFO:teuthology.orchestra.run.trial024.stdout:128849018913 2026-02-17T15:17:12.255 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018913 got 128849018913 for osd.3 2026-02-17T15:17:12.255 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:12.255 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:17:12.444 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:17:12.444 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:17:12.455 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2026-02-17T15:17:11.486917+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4999,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66534,"num_bytes_recovered":545350579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1415824,"kb_used_data":836700,"kb_used_omap":7,"kb_used_meta":579064,"kb_avail":281699696,"statfs":{"total":289910292480,"available":288460488704,"internally_reserved":0,"allocated":856780800,"data_stored":856869409,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7634,"internal_metadata":592962094},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8923,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8923,"num_bytes_recovered":73097216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001446"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":204681,"reported_epoch":34,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:17:09.639554+0000","last_change":"2026-02-17T15:15:40.196063+0000","last_active":"2026-02-17T15:17:09.639554+0000","last_peered":"2026-02-17T15:17:09.639554+0000","last_clean":"2026-02-17T15:15:34.564274+0000","last_became_active":"2026-02-17T15:15:40.195294+0000","last_became_peered":"2026-02-17T15:15:40.195294+0000","last_unstale":"2026-02-17T15:17:09.639554+0000","last_undegraded":"2026-02-17T15:17:09.639554+0000","last_fullsized":"2026-02-17T15:17:09.639554+0000","mapping_epoch":33,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4999,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66532,"num_bytes_recovered":545022387,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":56,"reported_epoch":34,"state":"active+clean","last_fresh":"2026-02-17T15:15:40.180214+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:15:40.180214+0000","last_peered":"2026-02-17T15:15:40.180214+0000","last_clean":"2026-02-17T15:15:40.180214+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:15:40.180214+0000","last_undegraded":"2026-02-17T15:15:40.180214+0000","last_fullsized":"2026-02-17T15:15:40.180214+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4999,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66532,"num_bytes_recovered":545022387,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1151442944,"data_stored":1151475712,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247857,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018913,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325708,"kb_used_data":286396,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046132,"statfs":{"total":96636764160,"available":96303239168,"internally_reserved":0,"allocated":293269504,"data_stored":293095435,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466084,"kb_used_data":286416,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905756,"statfs":{"total":96636764160,"available":96159494144,"internally_reserved":0,"allocated":293289984,"data_stored":293423115,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705721,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":624032,"kb_used_data":263888,"kb_used_omap":1,"kb_used_meta":360126,"kb_avail":93747808,"statfs":{"total":96636764160,"available":95997755392,"internally_reserved":0,"allocated":270221312,"data_stored":270350859,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":368769478},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":272498688,"data_stored":272506880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:17:12.456 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 204681, 'reported_epoch': 34, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:17:09.639554+0000', 'last_change': '2026-02-17T15:15:40.196063+0000', 'last_active': '2026-02-17T15:17:09.639554+0000', 'last_peered': '2026-02-17T15:17:09.639554+0000', 'last_clean': '2026-02-17T15:15:34.564274+0000', 'last_became_active': '2026-02-17T15:15:40.195294+0000', 'last_became_peered': '2026-02-17T15:15:40.195294+0000', 'last_unstale': '2026-02-17T15:17:09.639554+0000', 'last_undegraded': '2026-02-17T15:17:09.639554+0000', 'last_fullsized': '2026-02-17T15:17:09.639554+0000', 'mapping_epoch': 33, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 31, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4999, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 66532, 'num_bytes_recovered': 545022387, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:17:12.456 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:17:17.458 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:17:17.458 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:17:17.459 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:17:17.459 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:17:17.546 INFO:teuthology.orchestra.run.trial024.stdout:128849018915 2026-02-17T15:17:17.547 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:17.548 INFO:teuthology.orchestra.run.trial024.stdout:98784247859 2026-02-17T15:17:17.549 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:17.549 INFO:teuthology.orchestra.run.trial024.stdout:38654705724 2026-02-17T15:17:17.550 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:17.551 INFO:teuthology.orchestra.run.trial024.stdout:38654705723 2026-02-17T15:17:17.551 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:17.742 INFO:teuthology.orchestra.run.trial024.stdout:98784247858 2026-02-17T15:17:17.746 INFO:teuthology.orchestra.run.trial024.stdout:38654705722 2026-02-17T15:17:17.750 INFO:teuthology.orchestra.run.trial024.stdout:38654705723 2026-02-17T15:17:17.752 INFO:teuthology.orchestra.run.trial024.stdout:128849018914 2026-02-17T15:17:17.753 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247859 got 98784247858 for osd.1 2026-02-17T15:17:17.758 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705723 got 38654705722 for osd.2 2026-02-17T15:17:17.762 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705724 got 38654705723 for osd.0 2026-02-17T15:17:17.764 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018915 got 128849018914 for osd.3 2026-02-17T15:17:18.754 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:18.759 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:18.762 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:18.765 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:18.906 INFO:teuthology.orchestra.run.trial024.stdout:98784247858 2026-02-17T15:17:18.919 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247859 got 98784247858 for osd.1 2026-02-17T15:17:18.942 INFO:teuthology.orchestra.run.trial024.stdout:128849018914 2026-02-17T15:17:18.946 INFO:teuthology.orchestra.run.trial024.stdout:38654705722 2026-02-17T15:17:18.950 INFO:teuthology.orchestra.run.trial024.stdout:38654705723 2026-02-17T15:17:18.954 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018915 got 128849018914 for osd.3 2026-02-17T15:17:18.958 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705723 got 38654705722 for osd.2 2026-02-17T15:17:18.962 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705724 got 38654705723 for osd.0 2026-02-17T15:17:19.920 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:19.954 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:19.959 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:19.963 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:20.066 INFO:teuthology.orchestra.run.trial024.stdout:98784247859 2026-02-17T15:17:20.078 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247859 got 98784247859 for osd.1 2026-02-17T15:17:20.078 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:20.112 INFO:teuthology.orchestra.run.trial024.stdout:38654705724 2026-02-17T15:17:20.113 INFO:teuthology.orchestra.run.trial024.stdout:38654705723 2026-02-17T15:17:20.115 INFO:teuthology.orchestra.run.trial024.stdout:128849018916 2026-02-17T15:17:20.125 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705724 got 38654705724 for osd.0 2026-02-17T15:17:20.125 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:20.127 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705723 got 38654705723 for osd.2 2026-02-17T15:17:20.127 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:20.129 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018915 got 128849018916 for osd.3 2026-02-17T15:17:20.129 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:20.129 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:17:20.319 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:17:20.320 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:17:20.330 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2026-02-17T15:17:19.487912+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71532,"num_bytes_recovered":586294195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1434224,"kb_used_data":859252,"kb_used_omap":7,"kb_used_meta":574904,"kb_avail":281681296,"statfs":{"total":289910292480,"available":288441647104,"internally_reserved":0,"allocated":879874048,"data_stored":879948461,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7634,"internal_metadata":588702254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4999,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4998,"num_bytes_recovered":40943616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000994"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":214683,"reported_epoch":36,"state":"activating","last_fresh":"2026-02-17T15:17:17.567990+0000","last_change":"2026-02-17T15:17:17.568111+0000","last_active":"2026-02-17T15:17:16.565848+0000","last_peered":"2026-02-17T15:17:16.285737+0000","last_clean":"2026-02-17T15:17:16.285737+0000","last_became_active":"2026-02-17T15:15:40.195294+0000","last_became_peered":"2026-02-17T15:15:40.195294+0000","last_unstale":"2026-02-17T15:17:17.567990+0000","last_undegraded":"2026-02-17T15:17:17.567990+0000","last_fullsized":"2026-02-17T15:17:17.567990+0000","mapping_epoch":35,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71530,"num_bytes_recovered":585966003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":["3(0)","0(1)","1(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":71529}],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":58,"reported_epoch":35,"state":"active+clean","last_fresh":"2026-02-17T15:17:16.567539+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:17:16.567539+0000","last_peered":"2026-02-17T15:17:16.567539+0000","last_clean":"2026-02-17T15:17:16.567539+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:17:16.567539+0000","last_undegraded":"2026-02-17T15:17:16.567539+0000","last_fullsized":"2026-02-17T15:17:16.567539+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71530,"num_bytes_recovered":585966003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171922944,"data_stored":1171955712,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247859,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018916,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325720,"kb_used_data":286404,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046120,"statfs":{"total":96636764160,"available":96303226880,"internally_reserved":0,"allocated":293277696,"data_stored":293098895,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466092,"kb_used_data":286424,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905748,"statfs":{"total":96636764160,"available":96159485952,"internally_reserved":0,"allocated":293298176,"data_stored":293426575,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705724,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642412,"kb_used_data":286424,"kb_used_omap":1,"kb_used_meta":355966,"kb_avail":93729428,"statfs":{"total":96636764160,"available":95978934272,"internally_reserved":0,"allocated":293298176,"data_stored":293422991,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":364509638},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:17:20.331 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 214683, 'reported_epoch': 36, 'state': 'activating', 'last_fresh': '2026-02-17T15:17:17.567990+0000', 'last_change': '2026-02-17T15:17:17.568111+0000', 'last_active': '2026-02-17T15:17:16.565848+0000', 'last_peered': '2026-02-17T15:17:16.285737+0000', 'last_clean': '2026-02-17T15:17:16.285737+0000', 'last_became_active': '2026-02-17T15:15:40.195294+0000', 'last_became_peered': '2026-02-17T15:15:40.195294+0000', 'last_unstale': '2026-02-17T15:17:17.567990+0000', 'last_undegraded': '2026-02-17T15:17:17.567990+0000', 'last_fullsized': '2026-02-17T15:17:17.567990+0000', 'mapping_epoch': 35, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 34, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 71530, 'num_bytes_recovered': 585966003, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 0, 2], 'avail_no_missing': ['3(0)', '0(1)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [1, 2], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:17:20.331 DEBUG:tasks.ec_inconsistent_hinfo:not active 2026-02-17T15:17:25.332 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:17:25.333 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:17:25.333 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:17:25.334 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:17:25.419 INFO:teuthology.orchestra.run.trial024.stdout:128849018918 2026-02-17T15:17:25.420 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:25.421 INFO:teuthology.orchestra.run.trial024.stdout:38654705726 2026-02-17T15:17:25.422 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:25.423 INFO:teuthology.orchestra.run.trial024.stdout:98784247862 2026-02-17T15:17:25.423 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:25.429 INFO:teuthology.orchestra.run.trial024.stdout:38654705725 2026-02-17T15:17:25.430 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:25.587 INFO:teuthology.orchestra.run.trial024.stdout:128849018918 2026-02-17T15:17:25.600 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018918 got 128849018918 for osd.3 2026-02-17T15:17:25.600 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:25.618 INFO:teuthology.orchestra.run.trial024.stdout:98784247862 2026-02-17T15:17:25.619 INFO:teuthology.orchestra.run.trial024.stdout:38654705726 2026-02-17T15:17:25.625 INFO:teuthology.orchestra.run.trial024.stdout:38654705725 2026-02-17T15:17:25.630 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247862 got 98784247862 for osd.1 2026-02-17T15:17:25.631 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:25.632 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705726 got 38654705726 for osd.0 2026-02-17T15:17:25.632 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:25.637 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705725 got 38654705725 for osd.2 2026-02-17T15:17:25.638 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:25.638 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:17:25.822 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:17:25.822 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:17:25.834 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2026-02-17T15:17:25.488824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71532,"num_bytes_recovered":586294195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1434256,"kb_used_data":859276,"kb_used_omap":7,"kb_used_meta":574904,"kb_avail":281681264,"statfs":{"total":289910292480,"available":288441614336,"internally_reserved":0,"allocated":879898624,"data_stored":879958778,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7634,"internal_metadata":588702254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2766,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2765,"num_bytes_recovered":22650880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.924372"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":214690,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-17T15:17:17.570213+0000","last_change":"2026-02-17T15:17:17.570213+0000","last_active":"2026-02-17T15:17:17.570213+0000","last_peered":"2026-02-17T15:17:17.570213+0000","last_clean":"2026-02-17T15:17:17.570213+0000","last_became_active":"2026-02-17T15:17:17.569946+0000","last_became_peered":"2026-02-17T15:17:17.569946+0000","last_unstale":"2026-02-17T15:17:17.570213+0000","last_undegraded":"2026-02-17T15:17:17.570213+0000","last_fullsized":"2026-02-17T15:17:17.570213+0000","mapping_epoch":35,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T22:34:01.247241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71530,"num_bytes_recovered":585966003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":60,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-17T15:17:17.569403+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:17:17.569403+0000","last_peered":"2026-02-17T15:17:17.569403+0000","last_clean":"2026-02-17T15:17:17.569403+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:17:17.569403+0000","last_undegraded":"2026-02-17T15:17:17.569403+0000","last_fullsized":"2026-02-17T15:17:17.569403+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71530,"num_bytes_recovered":585966003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1170944000,"data_stored":1170976768,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247862,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325728,"kb_used_data":286412,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046112,"statfs":{"total":96636764160,"available":96303218688,"internally_reserved":0,"allocated":293285888,"data_stored":293102334,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466104,"kb_used_data":286432,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905736,"statfs":{"total":96636764160,"available":96159473664,"internally_reserved":0,"allocated":293306368,"data_stored":293430014,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705726,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642424,"kb_used_data":286432,"kb_used_omap":1,"kb_used_meta":355966,"kb_avail":93729416,"statfs":{"total":96636764160,"available":95978921984,"internally_reserved":0,"allocated":293306368,"data_stored":293426430,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":364509638},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291999744,"data_stored":292007936,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:17:25.835 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 214690, 'reported_epoch': 36, 'state': 'active+clean', 'last_fresh': '2026-02-17T15:17:17.570213+0000', 'last_change': '2026-02-17T15:17:17.570213+0000', 'last_active': '2026-02-17T15:17:17.570213+0000', 'last_peered': '2026-02-17T15:17:17.570213+0000', 'last_clean': '2026-02-17T15:17:17.570213+0000', 'last_became_active': '2026-02-17T15:17:17.569946+0000', 'last_became_peered': '2026-02-17T15:17:17.569946+0000', 'last_unstale': '2026-02-17T15:17:17.570213+0000', 'last_undegraded': '2026-02-17T15:17:17.570213+0000', 'last_fullsized': '2026-02-17T15:17:17.570213+0000', 'mapping_epoch': 35, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 36, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-18T22:34:01.247241+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 71530, 'num_bytes_recovered': 585966003, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:17:25.835 DEBUG:tasks.ec_inconsistent_hinfo:backfilled! 2026-02-17T15:17:25.835 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for clean 2026-02-17T15:17:25.835 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:17:25.987 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:17:25.987 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:17:25.997 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2026-02-17T15:17:25.488824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71532,"num_bytes_recovered":586294195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1434256,"kb_used_data":859276,"kb_used_omap":7,"kb_used_meta":574904,"kb_avail":281681264,"statfs":{"total":289910292480,"available":288441614336,"internally_reserved":0,"allocated":879898624,"data_stored":879958778,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7634,"internal_metadata":588702254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2766,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2765,"num_bytes_recovered":22650880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.924372"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":214690,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-17T15:17:17.570213+0000","last_change":"2026-02-17T15:17:17.570213+0000","last_active":"2026-02-17T15:17:17.570213+0000","last_peered":"2026-02-17T15:17:17.570213+0000","last_clean":"2026-02-17T15:17:17.570213+0000","last_became_active":"2026-02-17T15:17:17.569946+0000","last_became_peered":"2026-02-17T15:17:17.569946+0000","last_unstale":"2026-02-17T15:17:17.570213+0000","last_undegraded":"2026-02-17T15:17:17.570213+0000","last_fullsized":"2026-02-17T15:17:17.570213+0000","mapping_epoch":35,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T22:34:01.247241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71530,"num_bytes_recovered":585966003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":60,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-17T15:17:17.569403+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:17:17.569403+0000","last_peered":"2026-02-17T15:17:17.569403+0000","last_clean":"2026-02-17T15:17:17.569403+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:17:17.569403+0000","last_undegraded":"2026-02-17T15:17:17.569403+0000","last_fullsized":"2026-02-17T15:17:17.569403+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71530,"num_bytes_recovered":585966003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1170944000,"data_stored":1170976768,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247862,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325728,"kb_used_data":286412,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046112,"statfs":{"total":96636764160,"available":96303218688,"internally_reserved":0,"allocated":293285888,"data_stored":293102334,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466104,"kb_used_data":286432,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905736,"statfs":{"total":96636764160,"available":96159473664,"internally_reserved":0,"allocated":293306368,"data_stored":293430014,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705726,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642424,"kb_used_data":286432,"kb_used_omap":1,"kb_used_meta":355966,"kb_avail":93729416,"statfs":{"total":96636764160,"available":95978921984,"internally_reserved":0,"allocated":293306368,"data_stored":293426430,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":364509638},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291999744,"data_stored":292007936,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:17:25.998 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:17:26.149 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:17:26.149 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:17:26.159 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":138,"stamp":"2026-02-17T15:17:25.488824+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71532,"num_bytes_recovered":586294195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1434256,"kb_used_data":859276,"kb_used_omap":7,"kb_used_meta":574904,"kb_avail":281681264,"statfs":{"total":289910292480,"available":288441614336,"internally_reserved":0,"allocated":879898624,"data_stored":879958778,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7634,"internal_metadata":588702254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2766,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2765,"num_bytes_recovered":22650880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.924372"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":214690,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-17T15:17:17.570213+0000","last_change":"2026-02-17T15:17:17.570213+0000","last_active":"2026-02-17T15:17:17.570213+0000","last_peered":"2026-02-17T15:17:17.570213+0000","last_clean":"2026-02-17T15:17:17.570213+0000","last_became_active":"2026-02-17T15:17:17.569946+0000","last_became_peered":"2026-02-17T15:17:17.569946+0000","last_unstale":"2026-02-17T15:17:17.570213+0000","last_undegraded":"2026-02-17T15:17:17.570213+0000","last_fullsized":"2026-02-17T15:17:17.570213+0000","mapping_epoch":35,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T22:34:01.247241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71530,"num_bytes_recovered":585966003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":60,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-17T15:17:17.569403+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:17:17.569403+0000","last_peered":"2026-02-17T15:17:17.569403+0000","last_clean":"2026-02-17T15:17:17.569403+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:17:17.569403+0000","last_undegraded":"2026-02-17T15:17:17.569403+0000","last_fullsized":"2026-02-17T15:17:17.569403+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71530,"num_bytes_recovered":585966003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1170944000,"data_stored":1170976768,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247862,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018918,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325728,"kb_used_data":286412,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046112,"statfs":{"total":96636764160,"available":96303218688,"internally_reserved":0,"allocated":293285888,"data_stored":293102334,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466104,"kb_used_data":286432,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905736,"statfs":{"total":96636764160,"available":96159473664,"internally_reserved":0,"allocated":293306368,"data_stored":293430014,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705726,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642424,"kb_used_data":286432,"kb_used_omap":1,"kb_used_meta":355966,"kb_avail":93729416,"statfs":{"total":96636764160,"available":95978921984,"internally_reserved":0,"allocated":293306368,"data_stored":293426430,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":364509638},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291999744,"data_stored":292007936,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:17:26.160 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:clean! 2026-02-17T15:17:26.161 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:17:26.161 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:17:26.162 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:17:26.162 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:17:26.248 INFO:teuthology.orchestra.run.trial024.stdout:128849018919 2026-02-17T15:17:26.249 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:26.250 INFO:teuthology.orchestra.run.trial024.stdout:38654705727 2026-02-17T15:17:26.250 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:26.253 INFO:teuthology.orchestra.run.trial024.stdout:98784247863 2026-02-17T15:17:26.254 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:26.255 INFO:teuthology.orchestra.run.trial024.stdout:38654705726 2026-02-17T15:17:26.256 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:26.401 INFO:teuthology.orchestra.run.trial024.stdout:128849018918 2026-02-17T15:17:26.403 INFO:teuthology.orchestra.run.trial024.stdout:38654705726 2026-02-17T15:17:26.413 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018919 got 128849018918 for osd.3 2026-02-17T15:17:26.416 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705727 got 38654705726 for osd.0 2026-02-17T15:17:26.447 INFO:teuthology.orchestra.run.trial024.stdout:98784247862 2026-02-17T15:17:26.451 INFO:teuthology.orchestra.run.trial024.stdout:38654705725 2026-02-17T15:17:26.459 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247863 got 98784247862 for osd.1 2026-02-17T15:17:26.463 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705726 got 38654705725 for osd.2 2026-02-17T15:17:27.414 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:27.417 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:27.460 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:27.464 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:27.574 INFO:teuthology.orchestra.run.trial024.stdout:128849018919 2026-02-17T15:17:27.587 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018919 got 128849018919 for osd.3 2026-02-17T15:17:27.587 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:27.605 INFO:teuthology.orchestra.run.trial024.stdout:38654705728 2026-02-17T15:17:27.608 INFO:teuthology.orchestra.run.trial024.stdout:98784247863 2026-02-17T15:17:27.617 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705727 got 38654705728 for osd.0 2026-02-17T15:17:27.618 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:27.620 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247863 got 98784247863 for osd.1 2026-02-17T15:17:27.621 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:27.652 INFO:teuthology.orchestra.run.trial024.stdout:38654705727 2026-02-17T15:17:27.664 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705726 got 38654705727 for osd.2 2026-02-17T15:17:27.664 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:27.664 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:17:27.851 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:17:27.851 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:17:27.861 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2026-02-17T15:17:27.489073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71532,"num_bytes_recovered":586294195,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1434256,"kb_used_data":859276,"kb_used_omap":7,"kb_used_meta":574904,"kb_avail":281681264,"statfs":{"total":289910292480,"available":288441614336,"internally_reserved":0,"allocated":879898624,"data_stored":879958778,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":7634,"internal_metadata":588702254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2766,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2765,"num_bytes_recovered":22650880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001427"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":214690,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-17T15:17:17.570213+0000","last_change":"2026-02-17T15:17:17.570213+0000","last_active":"2026-02-17T15:17:17.570213+0000","last_peered":"2026-02-17T15:17:17.570213+0000","last_clean":"2026-02-17T15:17:17.570213+0000","last_became_active":"2026-02-17T15:17:17.569946+0000","last_became_peered":"2026-02-17T15:17:17.569946+0000","last_unstale":"2026-02-17T15:17:17.570213+0000","last_undegraded":"2026-02-17T15:17:17.570213+0000","last_fullsized":"2026-02-17T15:17:17.570213+0000","mapping_epoch":35,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T22:34:01.247241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71530,"num_bytes_recovered":585966003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":60,"reported_epoch":36,"state":"active+clean","last_fresh":"2026-02-17T15:17:17.569403+0000","last_change":"2026-02-17T15:15:38.941688+0000","last_active":"2026-02-17T15:17:17.569403+0000","last_peered":"2026-02-17T15:17:17.569403+0000","last_clean":"2026-02-17T15:17:17.569403+0000","last_became_active":"2026-02-17T15:15:38.937627+0000","last_became_peered":"2026-02-17T15:15:38.937627+0000","last_unstale":"2026-02-17T15:17:17.569403+0000","last_undegraded":"2026-02-17T15:17:17.569403+0000","last_fullsized":"2026-02-17T15:17:17.569403+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T01:30:35.949395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71530,"num_bytes_recovered":585966003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1170821120,"data_stored":1170853888,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":328192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":984064,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":0,"seq":98784247863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018919,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325728,"kb_used_data":286412,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046112,"statfs":{"total":96636764160,"available":96303218688,"internally_reserved":0,"allocated":293285888,"data_stored":293102334,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466104,"kb_used_data":286432,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905736,"statfs":{"total":96636764160,"available":96159473664,"internally_reserved":0,"allocated":293306368,"data_stored":293430014,"data_compressed":7497,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":9,"seq":38654705728,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":642424,"kb_used_data":286432,"kb_used_omap":1,"kb_used_meta":355966,"kb_avail":93729416,"statfs":{"total":96636764160,"available":95978921984,"internally_reserved":0,"allocated":293306368,"data_stored":293426430,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":1594,"internal_metadata":364509638},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":291876864,"data_stored":291885056,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:17:27.862 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 214690, 'reported_epoch': 36, 'state': 'active+clean', 'last_fresh': '2026-02-17T15:17:17.570213+0000', 'last_change': '2026-02-17T15:17:17.570213+0000', 'last_active': '2026-02-17T15:17:17.570213+0000', 'last_peered': '2026-02-17T15:17:17.570213+0000', 'last_clean': '2026-02-17T15:17:17.570213+0000', 'last_became_active': '2026-02-17T15:17:17.569946+0000', 'last_became_peered': '2026-02-17T15:17:17.569946+0000', 'last_unstale': '2026-02-17T15:17:17.570213+0000', 'last_undegraded': '2026-02-17T15:17:17.570213+0000', 'last_fullsized': '2026-02-17T15:17:17.570213+0000', 'mapping_epoch': 35, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 36, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-18T22:34:01.247241+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 71530, 'num_bytes_recovered': 585966003, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 0, 2], 'acting': [3, 0, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:17:27.862 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T15:17:28.042 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:17:28.042 INFO:teuthology.orchestra.run.trial024.stdout:{"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":220,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":36,"num_osds":4,"num_up_osds":4,"osd_up_since":1771341333,"num_in_osds":3,"osd_in_since":1771341337,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":2}],"num_pgs":2,"num_pools":2,"num_objects":71531,"data_bytes":586285866,"bytes_used":1468678144,"bytes_avail":288441614336,"bytes_total":289910292480,"recovering_objects_per_sec":276,"recovering_bytes_per_sec":2264764,"recovering_keys_per_sec":0,"num_objects_recovered":2765,"num_bytes_recovered":22650880,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2026-02-17T15:13:47:232255+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T15:16:33.482377+0000","services":{}},"progress_events":{}} 2026-02-17T15:17:28.055 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:{'fsid': 'ac9bccef-327a-4812-a379-08d174e6a4bc', 'health': {'status': 'HEALTH_OK', 'checks': {}, 'mutes': []}, 'election_epoch': 10, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 220, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 36, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1771341333, 'num_in_osds': 3, 'osd_in_since': 1771341337, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 2}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 71531, 'data_bytes': 586285866, 'bytes_used': 1468678144, 'bytes_avail': 288441614336, 'bytes_total': 289910292480, 'recovering_objects_per_sec': 276, 'recovering_bytes_per_sec': 2264764, 'recovering_keys_per_sec': 0, 'num_objects_recovered': 2765, 'num_bytes_recovered': 22650880, 'num_keys_recovered': 0}, 'fsmap': {'epoch': 1, 'btime': '2026-02-17T15:13:47:232255+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 5, 'modified': '2026-02-17T15:16:33.482377+0000', 'services': {}}, 'progress_events': {}} 2026-02-17T15:17:28.055 DEBUG:tasks.ec_inconsistent_hinfo:there are 0 unfound objects 2026-02-17T15:17:28.055 INFO:tasks.ec_inconsistent_hinfo:remove test object hash info from non-primary osd.0 shard and test backfill 2026-02-17T15:17:28.056 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:17:28.209 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:17:28.209 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":36,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:17:17.564092+0000","last_up_change":"2026-02-17T15:15:33.556007+0000","last_in_change":"2026-02-17T15:15:37.927186+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-17T15:14:06.508581+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":4268245710}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":4268245710}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":4268245710}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":4268245710}]},"public_addr":"10.20.193.24:6809/4268245710","cluster_addr":"10.20.193.24:6810/4268245710","heartbeat_back_addr":"10.20.193.24:6812/4268245710","heartbeat_front_addr":"10.20.193.24:6811/4268245710","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":19,"up_from":23,"up_thru":27,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":2457068227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":2457068227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":2457068227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":2457068227}]},"public_addr":"10.20.193.24:6805/2457068227","cluster_addr":"10.20.193.24:6806/2457068227","heartbeat_back_addr":"10.20.193.24:6808/2457068227","heartbeat_front_addr":"10.20.193.24:6807/2457068227","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":1012255776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":1012255776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":1012255776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":1012255776}]},"public_addr":"10.20.193.24:6813/1012255776","cluster_addr":"10.20.193.24:6814/1012255776","heartbeat_back_addr":"10.20.193.24:6816/1012255776","heartbeat_front_addr":"10.20.193.24:6815/1012255776","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":26,"up_from":30,"up_thru":35,"down_at":27,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":306838151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":306838151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":306838151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":306838151}]},"public_addr":"10.20.193.24:6801/306838151","cluster_addr":"10.20.193.24:6802/306838151","heartbeat_back_addr":"10.20.193.24:6804/306838151","heartbeat_front_addr":"10.20.193.24:6803/306838151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.138188+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2026-02-17T15:13:55.488216+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.278612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:54.537485+0000","dead_epoch":27}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:17:28.220 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-02-17T15:17:28.366 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:17:28.376 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":36,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,0,2],"up_primary":3,"acting":[3,0,2],"acting_primary":3} 2026-02-17T15:17:28.377 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-17T15:17:28.377 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T15:17:28.464 INFO:tasks.ceph.osd.0:Stopped 2026-02-17T15:17:28.464 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 "$object" rm-attr hinfo_key 2026-02-17T15:17:28.464 DEBUG:teuthology.orchestra.run.trial024:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 --op list |grep '"'"'"oid":"test"'"'"') 2026-02-17T15:17:28.464 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 "$object" rm-attr hinfo_key' 2026-02-17T15:17:28.468 INFO:teuthology.orchestra.run.trial024.stderr:+ '[' -z '' ']' 2026-02-17T15:17:28.468 INFO:teuthology.orchestra.run.trial024.stderr:+ return 2026-02-17T15:17:28.468 INFO:teuthology.orchestra.run.trial024.stderr:+ case $- in 2026-02-17T15:17:28.469 INFO:teuthology.orchestra.run.trial024.stderr:+ return 2026-02-17T15:17:28.469 INFO:teuthology.orchestra.run.trial024.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 --op list 2026-02-17T15:17:28.469 INFO:teuthology.orchestra.run.trial024.stderr:++ grep '"oid":"test"' 2026-02-17T15:17:30.858 INFO:teuthology.orchestra.run.trial024.stderr:+ object='["2.0s1",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":1,"max":0}]' 2026-02-17T15:17:30.858 INFO:teuthology.orchestra.run.trial024.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --pgid 2.0s1 '["2.0s1",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":1,"max":0}]' rm-attr hinfo_key 2026-02-17T15:17:34.773 INFO:tasks.ceph.osd.0:Restarting daemon 2026-02-17T15:17:34.773 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2026-02-17T15:17:34.814 INFO:tasks.ceph.osd.0:Started 2026-02-17T15:17:34.815 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-17T15:17:34.875 INFO:teuthology.orchestra.run.trial024.stderr:no valid command found; 10 closest matches: 2026-02-17T15:17:34.875 INFO:teuthology.orchestra.run.trial024.stderr:0 2026-02-17T15:17:34.875 INFO:teuthology.orchestra.run.trial024.stderr:1 2026-02-17T15:17:34.875 INFO:teuthology.orchestra.run.trial024.stderr:2 2026-02-17T15:17:34.875 INFO:teuthology.orchestra.run.trial024.stderr:abort 2026-02-17T15:17:34.875 INFO:teuthology.orchestra.run.trial024.stderr:assert 2026-02-17T15:17:34.875 INFO:teuthology.orchestra.run.trial024.stderr:config diff 2026-02-17T15:17:34.876 INFO:teuthology.orchestra.run.trial024.stderr:config diff get 2026-02-17T15:17:34.876 INFO:teuthology.orchestra.run.trial024.stderr:config get 2026-02-17T15:17:34.876 INFO:teuthology.orchestra.run.trial024.stderr:config help [] 2026-02-17T15:17:34.876 INFO:teuthology.orchestra.run.trial024.stderr:config set ... 2026-02-17T15:17:34.876 INFO:teuthology.orchestra.run.trial024.stderr:admin_socket: invalid command 2026-02-17T15:17:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-17T15:17:34.878 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-17T15:17:36.114 INFO:tasks.ceph.osd.0.trial024.stderr:2026-02-17T15:17:36.112+0000 7f613d2a18c0 -1 Falling back to public interface 2026-02-17T15:17:39.879 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-17T15:17:39.947 INFO:teuthology.orchestra.run.trial024.stderr:no valid command found; 10 closest matches: 2026-02-17T15:17:39.947 INFO:teuthology.orchestra.run.trial024.stderr:0 2026-02-17T15:17:39.947 INFO:teuthology.orchestra.run.trial024.stderr:1 2026-02-17T15:17:39.948 INFO:teuthology.orchestra.run.trial024.stderr:2 2026-02-17T15:17:39.948 INFO:teuthology.orchestra.run.trial024.stderr:abort 2026-02-17T15:17:39.948 INFO:teuthology.orchestra.run.trial024.stderr:assert 2026-02-17T15:17:39.948 INFO:teuthology.orchestra.run.trial024.stderr:config diff 2026-02-17T15:17:39.948 INFO:teuthology.orchestra.run.trial024.stderr:config diff get 2026-02-17T15:17:39.948 INFO:teuthology.orchestra.run.trial024.stderr:config get 2026-02-17T15:17:39.948 INFO:teuthology.orchestra.run.trial024.stderr:config help [] 2026-02-17T15:17:39.948 INFO:teuthology.orchestra.run.trial024.stderr:config set ... 2026-02-17T15:17:39.948 INFO:teuthology.orchestra.run.trial024.stderr:admin_socket: invalid command 2026-02-17T15:17:39.949 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-17T15:17:39.950 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2026-02-17T15:17:40.459 INFO:tasks.ceph.osd.0.trial024.stderr:2026-02-17T15:17:40.456+0000 7f613d2a18c0 -1 osd.0 36 log_to_monitors true 2026-02-17T15:17:40.874 INFO:tasks.ceph.osd.0.trial024.stderr:2026-02-17T15:17:40.872+0000 7f613107d640 -1 osd.0 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T15:17:44.951 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2026-02-17T15:17:45.024 INFO:teuthology.orchestra.run.trial024.stdout:{ 2026-02-17T15:17:45.024 INFO:teuthology.orchestra.run.trial024.stdout: "ops": [], 2026-02-17T15:17:45.024 INFO:teuthology.orchestra.run.trial024.stdout: "num_ops": 0 2026-02-17T15:17:45.024 INFO:teuthology.orchestra.run.trial024.stdout:} 2026-02-17T15:17:45.031 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.0 to be up 2026-02-17T15:17:45.032 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:17:45.172 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:17:45.173 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":41,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:17:42.877370+0000","last_up_change":"2026-02-17T15:17:41.874259+0000","last_in_change":"2026-02-17T15:15:37.927186+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-17T15:14:06.508581+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":36,"up_from":40,"up_thru":40,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":342414690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":342414690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":342414690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":342414690}]},"public_addr":"10.20.193.24:6809/342414690","cluster_addr":"10.20.193.24:6810/342414690","heartbeat_back_addr":"10.20.193.24:6812/342414690","heartbeat_front_addr":"10.20.193.24:6811/342414690","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":19,"up_from":23,"up_thru":27,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":2457068227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":2457068227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":2457068227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":2457068227}]},"public_addr":"10.20.193.24:6805/2457068227","cluster_addr":"10.20.193.24:6806/2457068227","heartbeat_back_addr":"10.20.193.24:6808/2457068227","heartbeat_front_addr":"10.20.193.24:6807/2457068227","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":1012255776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":1012255776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":1012255776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":1012255776}]},"public_addr":"10.20.193.24:6813/1012255776","cluster_addr":"10.20.193.24:6814/1012255776","heartbeat_back_addr":"10.20.193.24:6816/1012255776","heartbeat_front_addr":"10.20.193.24:6815/1012255776","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":26,"up_from":30,"up_thru":40,"down_at":27,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":306838151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":306838151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":306838151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":306838151}]},"public_addr":"10.20.193.24:6801/306838151","cluster_addr":"10.20.193.24:6802/306838151","heartbeat_back_addr":"10.20.193.24:6804/306838151","heartbeat_front_addr":"10.20.193.24:6803/306838151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.138188+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2026-02-17T15:13:55.488216+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.278612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:54.537485+0000","dead_epoch":27}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:17:45.183 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.0 is up 2026-02-17T15:17:45.184 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2026-02-17T15:17:45.332 INFO:tasks.ceph.mon.a.trial024.stderr:2026-02-17T15:17:45.328+0000 7fc1ce14a640 -1 mon.a@0(leader).osd e41 definitely_dead 0 2026-02-17T15:17:45.886 INFO:teuthology.orchestra.run.trial024.stderr:marked in osd.1. 2026-02-17T15:17:45.899 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 backfill from osd.0 to osd.1 complete 2026-02-17T15:17:50.900 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:17:50.901 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:17:50.901 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:17:50.902 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:17:50.988 INFO:teuthology.orchestra.run.trial024.stdout:38654705732 2026-02-17T15:17:50.988 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:50.990 INFO:teuthology.orchestra.run.trial024.stdout:171798691844 2026-02-17T15:17:50.991 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:50.997 INFO:teuthology.orchestra.run.trial024.stdout:98784247869 2026-02-17T15:17:50.997 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:51.005 INFO:teuthology.orchestra.run.trial024.stdout:128849018925 2026-02-17T15:17:51.006 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:51.155 INFO:teuthology.orchestra.run.trial024.stdout:38654705731 2026-02-17T15:17:51.167 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705732 got 38654705731 for osd.2 2026-02-17T15:17:51.173 INFO:teuthology.orchestra.run.trial024.stdout:171798691842 2026-02-17T15:17:51.187 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691844 got 171798691842 for osd.0 2026-02-17T15:17:51.213 INFO:teuthology.orchestra.run.trial024.stdout:128849018924 2026-02-17T15:17:51.213 INFO:teuthology.orchestra.run.trial024.stdout:98784247867 2026-02-17T15:17:51.226 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247869 got 98784247867 for osd.1 2026-02-17T15:17:51.228 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018925 got 128849018924 for osd.3 2026-02-17T15:17:52.168 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:52.188 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:52.228 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:52.229 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:52.318 INFO:teuthology.orchestra.run.trial024.stdout:38654705732 2026-02-17T15:17:52.330 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705732 got 38654705732 for osd.2 2026-02-17T15:17:52.331 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:52.364 INFO:teuthology.orchestra.run.trial024.stdout:171798691844 2026-02-17T15:17:52.377 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691844 got 171798691844 for osd.0 2026-02-17T15:17:52.377 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:52.406 INFO:teuthology.orchestra.run.trial024.stdout:98784247869 2026-02-17T15:17:52.410 INFO:teuthology.orchestra.run.trial024.stdout:128849018925 2026-02-17T15:17:52.419 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247869 got 98784247869 for osd.1 2026-02-17T15:17:52.419 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:52.422 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018925 got 128849018925 for osd.3 2026-02-17T15:17:52.423 DEBUG:teuthology.parallel:result is None 2026-02-17T15:17:52.423 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:17:52.614 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:17:52.615 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:17:52.625 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2026-02-17T15:17:51.492221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71950,"num_bytes_recovered":589717939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1592236,"kb_used_data":1145940,"kb_used_omap":14,"kb_used_meta":446193,"kb_avail":375895124,"statfs":{"total":386547056640,"available":384916606976,"internally_reserved":0,"allocated":1173442560,"data_stored":1173172464,"data_compressed":21474,"data_compressed_allocated":73728,"data_compressed_original":720896,"omap_allocated":14838,"internal_metadata":456902154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":3751936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000775"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":215558,"reported_epoch":44,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:17:48.594366+0000","last_change":"2026-02-17T15:17:47.891087+0000","last_active":"2026-02-17T15:17:48.594366+0000","last_peered":"2026-02-17T15:17:48.594366+0000","last_clean":"2026-02-17T15:17:42.882542+0000","last_became_active":"2026-02-17T15:17:47.890325+0000","last_became_peered":"2026-02-17T15:17:47.890325+0000","last_unstale":"2026-02-17T15:17:48.594366+0000","last_undegraded":"2026-02-17T15:17:48.594366+0000","last_fullsized":"2026-02-17T15:17:48.594366+0000","mapping_epoch":43,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71948,"num_bytes_recovered":589390259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,0,2],"avail_no_missing":["3(0)","0(1)","1(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":18,"reported_epoch":44,"state":"active+clean","last_fresh":"2026-02-17T15:17:47.887548+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:17:47.887548+0000","last_peered":"2026-02-17T15:17:47.887548+0000","last_clean":"2026-02-17T15:17:47.887548+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:17:47.887548+0000","last_undegraded":"2026-02-17T15:17:47.887548+0000","last_fullsized":"2026-02-17T15:17:47.887548+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71948,"num_bytes_recovered":589390259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171922944,"data_stored":1171955712,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470892,"kb_used_data":286492,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900948,"statfs":{"total":96636764160,"available":96154570752,"internally_reserved":0,"allocated":293367808,"data_stored":293454780,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018925,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325792,"kb_used_data":286476,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046048,"statfs":{"total":96636764160,"available":96303153152,"internally_reserved":0,"allocated":293351424,"data_stored":293131196,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466152,"kb_used_data":286476,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905688,"statfs":{"total":96636764160,"available":96159424512,"internally_reserved":0,"allocated":293351424,"data_stored":293131196,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691844,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329400,"kb_used_data":286496,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":94042440,"statfs":{"total":96636764160,"available":96299458560,"internally_reserved":0,"allocated":293371904,"data_stored":293455292,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:17:52.626 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 215558, 'reported_epoch': 44, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:17:48.594366+0000', 'last_change': '2026-02-17T15:17:47.891087+0000', 'last_active': '2026-02-17T15:17:48.594366+0000', 'last_peered': '2026-02-17T15:17:48.594366+0000', 'last_clean': '2026-02-17T15:17:42.882542+0000', 'last_became_active': '2026-02-17T15:17:47.890325+0000', 'last_became_peered': '2026-02-17T15:17:47.890325+0000', 'last_unstale': '2026-02-17T15:17:48.594366+0000', 'last_undegraded': '2026-02-17T15:17:48.594366+0000', 'last_fullsized': '2026-02-17T15:17:48.594366+0000', 'mapping_epoch': 43, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 41, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 71948, 'num_bytes_recovered': 589390259, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 2], 'acting': [3, 0, 2], 'avail_no_missing': ['3(0)', '0(1)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:17:52.626 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:17:57.628 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:17:57.628 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:17:57.629 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:17:57.629 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:17:57.717 INFO:teuthology.orchestra.run.trial024.stdout:98784247871 2026-02-17T15:17:57.717 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:57.720 INFO:teuthology.orchestra.run.trial024.stdout:38654705735 2026-02-17T15:17:57.720 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:57.723 INFO:teuthology.orchestra.run.trial024.stdout:171798691846 2026-02-17T15:17:57.724 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:57.738 INFO:teuthology.orchestra.run.trial024.stdout:128849018928 2026-02-17T15:17:57.739 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:57.877 INFO:teuthology.orchestra.run.trial024.stdout:38654705734 2026-02-17T15:17:57.882 INFO:teuthology.orchestra.run.trial024.stdout:98784247870 2026-02-17T15:17:57.885 INFO:teuthology.orchestra.run.trial024.stdout:171798691845 2026-02-17T15:17:57.889 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705735 got 38654705734 for osd.2 2026-02-17T15:17:57.895 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247871 got 98784247870 for osd.1 2026-02-17T15:17:57.897 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691846 got 171798691845 for osd.0 2026-02-17T15:17:57.931 INFO:teuthology.orchestra.run.trial024.stdout:128849018926 2026-02-17T15:17:57.943 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018928 got 128849018926 for osd.3 2026-02-17T15:17:58.891 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:17:58.895 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:17:58.898 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:17:58.943 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:17:59.050 INFO:teuthology.orchestra.run.trial024.stdout:38654705734 2026-02-17T15:17:59.062 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705735 got 38654705734 for osd.2 2026-02-17T15:17:59.087 INFO:teuthology.orchestra.run.trial024.stdout:171798691845 2026-02-17T15:17:59.089 INFO:teuthology.orchestra.run.trial024.stdout:98784247870 2026-02-17T15:17:59.099 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691846 got 171798691845 for osd.0 2026-02-17T15:17:59.102 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247871 got 98784247870 for osd.1 2026-02-17T15:17:59.141 INFO:teuthology.orchestra.run.trial024.stdout:128849018926 2026-02-17T15:17:59.155 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018928 got 128849018926 for osd.3 2026-02-17T15:18:00.063 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:18:00.100 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:18:00.102 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:18:00.155 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:18:00.209 INFO:teuthology.orchestra.run.trial024.stdout:38654705735 2026-02-17T15:18:00.222 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705735 got 38654705735 for osd.2 2026-02-17T15:18:00.222 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:00.254 INFO:teuthology.orchestra.run.trial024.stdout:98784247871 2026-02-17T15:18:00.265 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247871 got 98784247871 for osd.1 2026-02-17T15:18:00.266 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:00.272 INFO:teuthology.orchestra.run.trial024.stdout:171798691846 2026-02-17T15:18:00.284 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691846 got 171798691846 for osd.0 2026-02-17T15:18:00.284 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:00.303 INFO:teuthology.orchestra.run.trial024.stdout:128849018928 2026-02-17T15:18:00.314 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018928 got 128849018928 for osd.3 2026-02-17T15:18:00.315 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:00.315 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:18:00.502 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:18:00.502 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:18:00.512 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":164,"stamp":"2026-02-17T15:17:59.571276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71950,"num_bytes_recovered":589717939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1592236,"kb_used_data":1145940,"kb_used_omap":14,"kb_used_meta":446193,"kb_avail":375895124,"statfs":{"total":386547056640,"available":384916606976,"internally_reserved":0,"allocated":1173442560,"data_stored":1173172464,"data_compressed":21474,"data_compressed_allocated":73728,"data_compressed_original":720896,"omap_allocated":14838,"internal_metadata":456902154},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":3751936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.079305"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":215558,"reported_epoch":44,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:17:48.594366+0000","last_change":"2026-02-17T15:17:47.891087+0000","last_active":"2026-02-17T15:17:48.594366+0000","last_peered":"2026-02-17T15:17:48.594366+0000","last_clean":"2026-02-17T15:17:42.882542+0000","last_became_active":"2026-02-17T15:17:47.890325+0000","last_became_peered":"2026-02-17T15:17:47.890325+0000","last_unstale":"2026-02-17T15:17:48.594366+0000","last_undegraded":"2026-02-17T15:17:48.594366+0000","last_fullsized":"2026-02-17T15:17:48.594366+0000","mapping_epoch":43,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71948,"num_bytes_recovered":589390259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,0,2],"avail_no_missing":["3(0)","0(1)","1(1)","2(2)"],"object_location_counts":[{"shards":"0(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":18,"reported_epoch":44,"state":"active+clean","last_fresh":"2026-02-17T15:17:47.887548+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:17:47.887548+0000","last_peered":"2026-02-17T15:17:47.887548+0000","last_clean":"2026-02-17T15:17:47.887548+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:17:47.887548+0000","last_undegraded":"2026-02-17T15:17:47.887548+0000","last_fullsized":"2026-02-17T15:17:47.887548+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71948,"num_bytes_recovered":589390259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171922944,"data_stored":1171955712,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470892,"kb_used_data":286492,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900948,"statfs":{"total":96636764160,"available":96154570752,"internally_reserved":0,"allocated":293367808,"data_stored":293454780,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018928,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325792,"kb_used_data":286476,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046048,"statfs":{"total":96636764160,"available":96303153152,"internally_reserved":0,"allocated":293351424,"data_stored":293131196,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466152,"kb_used_data":286476,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905688,"statfs":{"total":96636764160,"available":96159424512,"internally_reserved":0,"allocated":293351424,"data_stored":293131196,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691846,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":329400,"kb_used_data":286496,"kb_used_omap":4,"kb_used_meta":42875,"kb_avail":94042440,"statfs":{"total":96636764160,"available":96299458560,"internally_reserved":0,"allocated":293371904,"data_stored":293455292,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4404,"internal_metadata":43904716},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:18:00.513 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 215558, 'reported_epoch': 44, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:17:48.594366+0000', 'last_change': '2026-02-17T15:17:47.891087+0000', 'last_active': '2026-02-17T15:17:48.594366+0000', 'last_peered': '2026-02-17T15:17:48.594366+0000', 'last_clean': '2026-02-17T15:17:42.882542+0000', 'last_became_active': '2026-02-17T15:17:47.890325+0000', 'last_became_peered': '2026-02-17T15:17:47.890325+0000', 'last_unstale': '2026-02-17T15:17:48.594366+0000', 'last_undegraded': '2026-02-17T15:17:48.594366+0000', 'last_fullsized': '2026-02-17T15:17:48.594366+0000', 'mapping_epoch': 43, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 41, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 71948, 'num_bytes_recovered': 589390259, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 2], 'acting': [3, 0, 2], 'avail_no_missing': ['3(0)', '0(1)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '0(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:18:00.513 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:18:05.514 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:18:05.515 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:18:05.516 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:18:05.516 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:18:05.604 INFO:teuthology.orchestra.run.trial024.stdout:128849018930 2026-02-17T15:18:05.605 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:18:05.606 INFO:teuthology.orchestra.run.trial024.stdout:171798691849 2026-02-17T15:18:05.606 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:18:05.612 INFO:teuthology.orchestra.run.trial024.stdout:38654705737 2026-02-17T15:18:05.612 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:18:05.613 INFO:teuthology.orchestra.run.trial024.stdout:98784247874 2026-02-17T15:18:05.614 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:18:05.757 INFO:teuthology.orchestra.run.trial024.stdout:171798691847 2026-02-17T15:18:05.769 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691849 got 171798691847 for osd.0 2026-02-17T15:18:05.773 INFO:teuthology.orchestra.run.trial024.stdout:128849018929 2026-02-17T15:18:05.786 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018930 got 128849018929 for osd.3 2026-02-17T15:18:05.806 INFO:teuthology.orchestra.run.trial024.stdout:38654705736 2026-02-17T15:18:05.818 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705737 got 38654705736 for osd.2 2026-02-17T15:18:05.825 INFO:teuthology.orchestra.run.trial024.stdout:98784247873 2026-02-17T15:18:05.837 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247874 got 98784247873 for osd.1 2026-02-17T15:18:06.770 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:18:06.787 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:18:06.818 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:18:06.838 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:18:06.923 INFO:teuthology.orchestra.run.trial024.stdout:171798691847 2026-02-17T15:18:06.936 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691849 got 171798691847 for osd.0 2026-02-17T15:18:06.958 INFO:teuthology.orchestra.run.trial024.stdout:128849018929 2026-02-17T15:18:06.970 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018930 got 128849018929 for osd.3 2026-02-17T15:18:07.011 INFO:teuthology.orchestra.run.trial024.stdout:98784247873 2026-02-17T15:18:07.013 INFO:teuthology.orchestra.run.trial024.stdout:38654705736 2026-02-17T15:18:07.022 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247874 got 98784247873 for osd.1 2026-02-17T15:18:07.025 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705737 got 38654705736 for osd.2 2026-02-17T15:18:07.937 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:18:07.972 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:18:08.023 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:18:08.026 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:18:08.082 INFO:teuthology.orchestra.run.trial024.stdout:171798691849 2026-02-17T15:18:08.094 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691849 got 171798691849 for osd.0 2026-02-17T15:18:08.095 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:08.128 INFO:teuthology.orchestra.run.trial024.stdout:128849018930 2026-02-17T15:18:08.140 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018930 got 128849018930 for osd.3 2026-02-17T15:18:08.140 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:08.176 INFO:teuthology.orchestra.run.trial024.stdout:98784247874 2026-02-17T15:18:08.180 INFO:teuthology.orchestra.run.trial024.stdout:38654705738 2026-02-17T15:18:08.188 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247874 got 98784247874 for osd.1 2026-02-17T15:18:08.188 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:08.192 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705737 got 38654705738 for osd.2 2026-02-17T15:18:08.192 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:08.192 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:18:08.384 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:18:08.385 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:18:08.396 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2026-02-17T15:18:07.494221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71950,"num_bytes_recovered":589717939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1314508,"kb_used_data":859892,"kb_used_omap":14,"kb_used_meta":454513,"kb_avail":376172852,"statfs":{"total":386547056640,"available":385201000448,"internally_reserved":0,"allocated":880529408,"data_stored":880213180,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":14838,"internal_metadata":465421834},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001008"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":215571,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-17T15:18:00.575515+0000","last_change":"2026-02-17T15:18:00.575515+0000","last_active":"2026-02-17T15:18:00.575515+0000","last_peered":"2026-02-17T15:18:00.575515+0000","last_clean":"2026-02-17T15:18:00.575515+0000","last_became_active":"2026-02-17T15:18:00.575206+0000","last_became_peered":"2026-02-17T15:18:00.575206+0000","last_unstale":"2026-02-17T15:18:00.575515+0000","last_undegraded":"2026-02-17T15:18:00.575515+0000","last_fullsized":"2026-02-17T15:18:00.575515+0000","mapping_epoch":45,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T19:12:24.877703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71948,"num_bytes_recovered":589390259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":22,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-17T15:18:00.574520+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:00.574520+0000","last_peered":"2026-02-17T15:18:00.574520+0000","last_clean":"2026-02-17T15:18:00.574520+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:00.574520+0000","last_undegraded":"2026-02-17T15:18:00.574520+0000","last_fullsized":"2026-02-17T15:18:00.574520+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71948,"num_bytes_recovered":589390259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171922944,"data_stored":1171955712,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470916,"kb_used_data":286508,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900924,"statfs":{"total":96636764160,"available":96154546176,"internally_reserved":0,"allocated":293384192,"data_stored":293461679,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325812,"kb_used_data":286492,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046028,"statfs":{"total":96636764160,"available":96303132672,"internally_reserved":0,"allocated":293367808,"data_stored":293138095,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466164,"kb_used_data":286492,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905676,"statfs":{"total":96636764160,"available":96159412224,"internally_reserved":0,"allocated":293367808,"data_stored":293138095,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51616,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":51195,"kb_avail":94320224,"statfs":{"total":96636764160,"available":96583909376,"internally_reserved":0,"allocated":409600,"data_stored":475311,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":4404,"internal_metadata":52424396},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:18:08.397 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 215571, 'reported_epoch': 46, 'state': 'active+clean', 'last_fresh': '2026-02-17T15:18:00.575515+0000', 'last_change': '2026-02-17T15:18:00.575515+0000', 'last_active': '2026-02-17T15:18:00.575515+0000', 'last_peered': '2026-02-17T15:18:00.575515+0000', 'last_clean': '2026-02-17T15:18:00.575515+0000', 'last_became_active': '2026-02-17T15:18:00.575206+0000', 'last_became_peered': '2026-02-17T15:18:00.575206+0000', 'last_unstale': '2026-02-17T15:18:00.575515+0000', 'last_undegraded': '2026-02-17T15:18:00.575515+0000', 'last_fullsized': '2026-02-17T15:18:00.575515+0000', 'mapping_epoch': 45, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 46, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-18T19:12:24.877703+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 71948, 'num_bytes_recovered': 589390259, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:18:08.397 DEBUG:tasks.ec_inconsistent_hinfo:backfilled! 2026-02-17T15:18:08.397 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for clean 2026-02-17T15:18:08.398 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:18:08.543 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:18:08.543 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:18:08.554 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2026-02-17T15:18:07.494221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71950,"num_bytes_recovered":589717939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1314508,"kb_used_data":859892,"kb_used_omap":14,"kb_used_meta":454513,"kb_avail":376172852,"statfs":{"total":386547056640,"available":385201000448,"internally_reserved":0,"allocated":880529408,"data_stored":880213180,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":14838,"internal_metadata":465421834},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001008"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":215571,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-17T15:18:00.575515+0000","last_change":"2026-02-17T15:18:00.575515+0000","last_active":"2026-02-17T15:18:00.575515+0000","last_peered":"2026-02-17T15:18:00.575515+0000","last_clean":"2026-02-17T15:18:00.575515+0000","last_became_active":"2026-02-17T15:18:00.575206+0000","last_became_peered":"2026-02-17T15:18:00.575206+0000","last_unstale":"2026-02-17T15:18:00.575515+0000","last_undegraded":"2026-02-17T15:18:00.575515+0000","last_fullsized":"2026-02-17T15:18:00.575515+0000","mapping_epoch":45,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T19:12:24.877703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71948,"num_bytes_recovered":589390259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":22,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-17T15:18:00.574520+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:00.574520+0000","last_peered":"2026-02-17T15:18:00.574520+0000","last_clean":"2026-02-17T15:18:00.574520+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:00.574520+0000","last_undegraded":"2026-02-17T15:18:00.574520+0000","last_fullsized":"2026-02-17T15:18:00.574520+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71948,"num_bytes_recovered":589390259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171922944,"data_stored":1171955712,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470916,"kb_used_data":286508,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900924,"statfs":{"total":96636764160,"available":96154546176,"internally_reserved":0,"allocated":293384192,"data_stored":293461679,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325812,"kb_used_data":286492,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046028,"statfs":{"total":96636764160,"available":96303132672,"internally_reserved":0,"allocated":293367808,"data_stored":293138095,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466164,"kb_used_data":286492,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905676,"statfs":{"total":96636764160,"available":96159412224,"internally_reserved":0,"allocated":293367808,"data_stored":293138095,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51616,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":51195,"kb_avail":94320224,"statfs":{"total":96636764160,"available":96583909376,"internally_reserved":0,"allocated":409600,"data_stored":475311,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":4404,"internal_metadata":52424396},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:18:08.555 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:18:08.700 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:18:08.700 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:18:08.710 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2026-02-17T15:18:07.494221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71950,"num_bytes_recovered":589717939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1314508,"kb_used_data":859892,"kb_used_omap":14,"kb_used_meta":454513,"kb_avail":376172852,"statfs":{"total":386547056640,"available":385201000448,"internally_reserved":0,"allocated":880529408,"data_stored":880213180,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":14838,"internal_metadata":465421834},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001008"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":215571,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-17T15:18:00.575515+0000","last_change":"2026-02-17T15:18:00.575515+0000","last_active":"2026-02-17T15:18:00.575515+0000","last_peered":"2026-02-17T15:18:00.575515+0000","last_clean":"2026-02-17T15:18:00.575515+0000","last_became_active":"2026-02-17T15:18:00.575206+0000","last_became_peered":"2026-02-17T15:18:00.575206+0000","last_unstale":"2026-02-17T15:18:00.575515+0000","last_undegraded":"2026-02-17T15:18:00.575515+0000","last_fullsized":"2026-02-17T15:18:00.575515+0000","mapping_epoch":45,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T19:12:24.877703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71948,"num_bytes_recovered":589390259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":22,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-17T15:18:00.574520+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:00.574520+0000","last_peered":"2026-02-17T15:18:00.574520+0000","last_clean":"2026-02-17T15:18:00.574520+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:00.574520+0000","last_undegraded":"2026-02-17T15:18:00.574520+0000","last_fullsized":"2026-02-17T15:18:00.574520+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71948,"num_bytes_recovered":589390259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171922944,"data_stored":1171955712,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470916,"kb_used_data":286508,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900924,"statfs":{"total":96636764160,"available":96154546176,"internally_reserved":0,"allocated":293384192,"data_stored":293461679,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325812,"kb_used_data":286492,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046028,"statfs":{"total":96636764160,"available":96303132672,"internally_reserved":0,"allocated":293367808,"data_stored":293138095,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466164,"kb_used_data":286492,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905676,"statfs":{"total":96636764160,"available":96159412224,"internally_reserved":0,"allocated":293367808,"data_stored":293138095,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691849,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51616,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":51195,"kb_avail":94320224,"statfs":{"total":96636764160,"available":96583909376,"internally_reserved":0,"allocated":409600,"data_stored":475311,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":4404,"internal_metadata":52424396},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:18:08.711 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:clean! 2026-02-17T15:18:08.712 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:18:08.712 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:18:08.713 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:18:08.713 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:18:08.800 INFO:teuthology.orchestra.run.trial024.stdout:38654705739 2026-02-17T15:18:08.800 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:18:08.801 INFO:teuthology.orchestra.run.trial024.stdout:171798691850 2026-02-17T15:18:08.802 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:18:08.805 INFO:teuthology.orchestra.run.trial024.stdout:98784247875 2026-02-17T15:18:08.806 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:18:08.807 INFO:teuthology.orchestra.run.trial024.stdout:128849018932 2026-02-17T15:18:08.808 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:18:08.999 INFO:teuthology.orchestra.run.trial024.stdout:98784247874 2026-02-17T15:18:09.000 INFO:teuthology.orchestra.run.trial024.stdout:38654705738 2026-02-17T15:18:09.004 INFO:teuthology.orchestra.run.trial024.stdout:171798691849 2026-02-17T15:18:09.012 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247875 got 98784247874 for osd.1 2026-02-17T15:18:09.013 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705739 got 38654705738 for osd.2 2026-02-17T15:18:09.016 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691850 got 171798691849 for osd.0 2026-02-17T15:18:09.040 INFO:teuthology.orchestra.run.trial024.stdout:128849018930 2026-02-17T15:18:09.052 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018932 got 128849018930 for osd.3 2026-02-17T15:18:10.013 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:18:10.013 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:18:10.017 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:18:10.053 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:18:10.162 INFO:teuthology.orchestra.run.trial024.stdout:98784247875 2026-02-17T15:18:10.167 INFO:teuthology.orchestra.run.trial024.stdout:38654705739 2026-02-17T15:18:10.174 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247875 got 98784247875 for osd.1 2026-02-17T15:18:10.174 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:10.180 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 38654705739 got 38654705739 for osd.2 2026-02-17T15:18:10.180 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:10.202 INFO:teuthology.orchestra.run.trial024.stdout:171798691850 2026-02-17T15:18:10.207 INFO:teuthology.orchestra.run.trial024.stdout:128849018932 2026-02-17T15:18:10.213 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691850 got 171798691850 for osd.0 2026-02-17T15:18:10.214 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:10.219 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 128849018932 got 128849018932 for osd.3 2026-02-17T15:18:10.219 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:10.219 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:18:10.407 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:18:10.408 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:18:10.418 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2026-02-17T15:18:09.494541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71950,"num_bytes_recovered":589717939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1314508,"kb_used_data":859892,"kb_used_omap":14,"kb_used_meta":454513,"kb_avail":376172852,"statfs":{"total":386547056640,"available":385201000448,"internally_reserved":0,"allocated":880529408,"data_stored":880213180,"data_compressed":17214,"data_compressed_allocated":65536,"data_compressed_original":704512,"omap_allocated":14838,"internal_metadata":465421834},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.923264"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":215571,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-17T15:18:00.575515+0000","last_change":"2026-02-17T15:18:00.575515+0000","last_active":"2026-02-17T15:18:00.575515+0000","last_peered":"2026-02-17T15:18:00.575515+0000","last_clean":"2026-02-17T15:18:00.575515+0000","last_became_active":"2026-02-17T15:18:00.575206+0000","last_became_peered":"2026-02-17T15:18:00.575206+0000","last_unstale":"2026-02-17T15:18:00.575515+0000","last_undegraded":"2026-02-17T15:18:00.575515+0000","last_fullsized":"2026-02-17T15:18:00.575515+0000","mapping_epoch":45,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T19:12:24.877703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71948,"num_bytes_recovered":589390259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":22,"reported_epoch":46,"state":"active+clean","last_fresh":"2026-02-17T15:18:00.574520+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:00.574520+0000","last_peered":"2026-02-17T15:18:00.574520+0000","last_clean":"2026-02-17T15:18:00.574520+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:00.574520+0000","last_undegraded":"2026-02-17T15:18:00.574520+0000","last_fullsized":"2026-02-17T15:18:00.574520+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71948,"num_bytes_recovered":589390259,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171922944,"data_stored":1171955712,"data_compressed":20288,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247875,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":470916,"kb_used_data":286508,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900924,"statfs":{"total":96636764160,"available":96154546176,"internally_reserved":0,"allocated":293384192,"data_stored":293461679,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":30,"seq":128849018932,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":325812,"kb_used_data":286492,"kb_used_omap":4,"kb_used_meta":39291,"kb_avail":94046028,"statfs":{"total":96636764160,"available":96303132672,"internally_reserved":0,"allocated":293367808,"data_stored":293138095,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":4446,"internal_metadata":40234658},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":466164,"kb_used_data":286492,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93905676,"statfs":{"total":96636764160,"available":96159412224,"internally_reserved":0,"allocated":293367808,"data_stored":293138095,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691850,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51616,"kb_used_data":400,"kb_used_omap":4,"kb_used_meta":51195,"kb_avail":94320224,"statfs":{"total":96636764160,"available":96583909376,"internally_reserved":0,"allocated":409600,"data_stored":475311,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":4404,"internal_metadata":52424396},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:18:10.419 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 215571, 'reported_epoch': 46, 'state': 'active+clean', 'last_fresh': '2026-02-17T15:18:00.575515+0000', 'last_change': '2026-02-17T15:18:00.575515+0000', 'last_active': '2026-02-17T15:18:00.575515+0000', 'last_peered': '2026-02-17T15:18:00.575515+0000', 'last_clean': '2026-02-17T15:18:00.575515+0000', 'last_became_active': '2026-02-17T15:18:00.575206+0000', 'last_became_peered': '2026-02-17T15:18:00.575206+0000', 'last_unstale': '2026-02-17T15:18:00.575515+0000', 'last_undegraded': '2026-02-17T15:18:00.575515+0000', 'last_fullsized': '2026-02-17T15:18:00.575515+0000', 'mapping_epoch': 45, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 46, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2026-02-18T19:12:24.877703+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 71948, 'num_bytes_recovered': 589390259, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 2], 'acting': [3, 1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:18:10.419 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T15:18:10.596 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:18:10.596 INFO:teuthology.orchestra.run.trial024.stdout:{"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":263,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":46,"num_osds":4,"num_up_osds":4,"osd_up_since":1771341461,"num_in_osds":4,"osd_in_since":1771341465,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":2}],"num_pgs":2,"num_pools":2,"num_objects":71531,"data_bytes":586285866,"bytes_used":1346056192,"bytes_avail":385201000448,"bytes_total":386547056640},"fsmap":{"epoch":1,"btime":"2026-02-17T15:13:47:232255+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2026-02-17T15:16:33.482377+0000","services":{}},"progress_events":{}} 2026-02-17T15:18:10.608 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:{'fsid': 'ac9bccef-327a-4812-a379-08d174e6a4bc', 'health': {'status': 'HEALTH_OK', 'checks': {}, 'mutes': []}, 'election_epoch': 10, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 263, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 46, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1771341461, 'num_in_osds': 4, 'osd_in_since': 1771341465, 'num_remapped_pgs': 0}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 2}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 71531, 'data_bytes': 586285866, 'bytes_used': 1346056192, 'bytes_avail': 385201000448, 'bytes_total': 386547056640}, 'fsmap': {'epoch': 1, 'btime': '2026-02-17T15:13:47:232255+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 5, 'modified': '2026-02-17T15:16:33.482377+0000', 'services': {}}, 'progress_events': {}} 2026-02-17T15:18:10.608 DEBUG:tasks.ec_inconsistent_hinfo:there are 0 unfound objects 2026-02-17T15:18:10.608 INFO:tasks.ec_inconsistent_hinfo:remove hash info from two shards and test backfill 2026-02-17T15:18:10.608 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:18:10.762 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:18:10.763 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":46,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:18:00.570792+0000","last_up_change":"2026-02-17T15:17:41.874259+0000","last_in_change":"2026-02-17T15:17:45.881799+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":9,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-17T15:14:06.508581+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":36,"up_from":40,"up_thru":40,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":342414690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":342414690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":342414690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":342414690}]},"public_addr":"10.20.193.24:6809/342414690","cluster_addr":"10.20.193.24:6810/342414690","heartbeat_back_addr":"10.20.193.24:6812/342414690","heartbeat_front_addr":"10.20.193.24:6811/342414690","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":19,"up_from":23,"up_thru":42,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":2457068227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":2457068227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":2457068227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":2457068227}]},"public_addr":"10.20.193.24:6805/2457068227","cluster_addr":"10.20.193.24:6806/2457068227","heartbeat_back_addr":"10.20.193.24:6808/2457068227","heartbeat_front_addr":"10.20.193.24:6807/2457068227","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":1012255776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":1012255776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":1012255776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":1012255776}]},"public_addr":"10.20.193.24:6813/1012255776","cluster_addr":"10.20.193.24:6814/1012255776","heartbeat_back_addr":"10.20.193.24:6816/1012255776","heartbeat_front_addr":"10.20.193.24:6815/1012255776","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":26,"up_from":30,"up_thru":45,"down_at":27,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":306838151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":306838151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":306838151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":306838151}]},"public_addr":"10.20.193.24:6801/306838151","cluster_addr":"10.20.193.24:6802/306838151","heartbeat_back_addr":"10.20.193.24:6804/306838151","heartbeat_front_addr":"10.20.193.24:6803/306838151","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.138188+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.488216+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.278612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:54.537485+0000","dead_epoch":27}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:18:10.774 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-02-17T15:18:10.928 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:18:10.939 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":46,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,1,2],"up_primary":3,"acting":[3,1,2],"acting_primary":3} 2026-02-17T15:18:10.940 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-17T15:18:10.940 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T15:18:11.925 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2026-02-17T15:18:17.042 INFO:tasks.ceph.osd.3:Stopped 2026-02-17T15:18:17.042 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 "$object" rm-attr hinfo_key 2026-02-17T15:18:17.042 DEBUG:teuthology.orchestra.run.trial024:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list |grep '"'"'"oid":"test"'"'"') 2026-02-17T15:18:17.042 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 "$object" rm-attr hinfo_key' 2026-02-17T15:18:17.048 INFO:teuthology.orchestra.run.trial024.stderr:+ '[' -z '' ']' 2026-02-17T15:18:17.048 INFO:teuthology.orchestra.run.trial024.stderr:+ return 2026-02-17T15:18:17.048 INFO:teuthology.orchestra.run.trial024.stderr:+ case $- in 2026-02-17T15:18:17.048 INFO:teuthology.orchestra.run.trial024.stderr:+ return 2026-02-17T15:18:17.049 INFO:teuthology.orchestra.run.trial024.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 --op list 2026-02-17T15:18:17.049 INFO:teuthology.orchestra.run.trial024.stderr:++ grep '"oid":"test"' 2026-02-17T15:18:17.629 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2026-02-17T15:18:18.934 INFO:teuthology.orchestra.run.trial024.stderr:+ object='["2.0s0",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":0,"max":0}]' 2026-02-17T15:18:18.935 INFO:teuthology.orchestra.run.trial024.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --pgid 2.0s0 '["2.0s0",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":0,"max":0}]' rm-attr hinfo_key 2026-02-17T15:18:21.717 INFO:tasks.ceph.osd.3:Restarting daemon 2026-02-17T15:18:21.717 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2026-02-17T15:18:21.762 INFO:tasks.ceph.osd.3:Started 2026-02-17T15:18:21.762 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-02-17T15:18:21.820 INFO:teuthology.orchestra.run.trial024.stderr:no valid command found; 10 closest matches: 2026-02-17T15:18:21.820 INFO:teuthology.orchestra.run.trial024.stderr:0 2026-02-17T15:18:21.821 INFO:teuthology.orchestra.run.trial024.stderr:1 2026-02-17T15:18:21.821 INFO:teuthology.orchestra.run.trial024.stderr:2 2026-02-17T15:18:21.821 INFO:teuthology.orchestra.run.trial024.stderr:abort 2026-02-17T15:18:21.821 INFO:teuthology.orchestra.run.trial024.stderr:assert 2026-02-17T15:18:21.821 INFO:teuthology.orchestra.run.trial024.stderr:config diff 2026-02-17T15:18:21.821 INFO:teuthology.orchestra.run.trial024.stderr:config diff get 2026-02-17T15:18:21.821 INFO:teuthology.orchestra.run.trial024.stderr:config get 2026-02-17T15:18:21.821 INFO:teuthology.orchestra.run.trial024.stderr:config help [] 2026-02-17T15:18:21.821 INFO:teuthology.orchestra.run.trial024.stderr:config set ... 2026-02-17T15:18:21.821 INFO:teuthology.orchestra.run.trial024.stderr:admin_socket: invalid command 2026-02-17T15:18:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-17T15:18:21.823 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2026-02-17T15:18:23.026 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:18:23.024+0000 7fd34912e8c0 -1 Falling back to public interface 2026-02-17T15:18:26.824 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-02-17T15:18:26.894 INFO:teuthology.orchestra.run.trial024.stderr:no valid command found; 10 closest matches: 2026-02-17T15:18:26.895 INFO:teuthology.orchestra.run.trial024.stderr:0 2026-02-17T15:18:26.895 INFO:teuthology.orchestra.run.trial024.stderr:1 2026-02-17T15:18:26.895 INFO:teuthology.orchestra.run.trial024.stderr:2 2026-02-17T15:18:26.895 INFO:teuthology.orchestra.run.trial024.stderr:abort 2026-02-17T15:18:26.895 INFO:teuthology.orchestra.run.trial024.stderr:assert 2026-02-17T15:18:26.895 INFO:teuthology.orchestra.run.trial024.stderr:bluestore allocator dump block 2026-02-17T15:18:26.895 INFO:teuthology.orchestra.run.trial024.stderr:bluestore allocator fragmentation block 2026-02-17T15:18:26.895 INFO:teuthology.orchestra.run.trial024.stderr:bluestore allocator fragmentation histogram block [] [] 2026-02-17T15:18:26.895 INFO:teuthology.orchestra.run.trial024.stderr:bluestore allocator score block 2026-02-17T15:18:26.895 INFO:teuthology.orchestra.run.trial024.stderr:config diff 2026-02-17T15:18:26.896 INFO:teuthology.orchestra.run.trial024.stderr:admin_socket: invalid command 2026-02-17T15:18:26.896 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-17T15:18:26.897 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2026-02-17T15:18:27.162 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:18:27.160+0000 7fd34912e8c0 -1 osd.3 46 log_to_monitors true 2026-02-17T15:18:27.222 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:18:27.220+0000 7fd33cf0a640 -1 osd.3 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T15:18:31.898 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2026-02-17T15:18:31.962 INFO:teuthology.orchestra.run.trial024.stdout:{ 2026-02-17T15:18:31.963 INFO:teuthology.orchestra.run.trial024.stdout: "ops": [], 2026-02-17T15:18:31.963 INFO:teuthology.orchestra.run.trial024.stdout: "num_ops": 0 2026-02-17T15:18:31.963 INFO:teuthology.orchestra.run.trial024.stdout:} 2026-02-17T15:18:31.970 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.3 to be up 2026-02-17T15:18:31.970 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:18:32.123 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:18:32.124 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":51,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:18:29.220366+0000","last_up_change":"2026-02-17T15:18:28.217975+0000","last_in_change":"2026-02-17T15:17:45.881799+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-17T15:14:06.508581+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":36,"up_from":40,"up_thru":40,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":342414690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":342414690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":342414690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":342414690}]},"public_addr":"10.20.193.24:6809/342414690","cluster_addr":"10.20.193.24:6810/342414690","heartbeat_back_addr":"10.20.193.24:6812/342414690","heartbeat_front_addr":"10.20.193.24:6811/342414690","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":19,"up_from":23,"up_thru":47,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":2457068227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":2457068227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":2457068227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":2457068227}]},"public_addr":"10.20.193.24:6805/2457068227","cluster_addr":"10.20.193.24:6806/2457068227","heartbeat_back_addr":"10.20.193.24:6808/2457068227","heartbeat_front_addr":"10.20.193.24:6807/2457068227","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":1012255776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":1012255776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":1012255776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":1012255776}]},"public_addr":"10.20.193.24:6813/1012255776","cluster_addr":"10.20.193.24:6814/1012255776","heartbeat_back_addr":"10.20.193.24:6816/1012255776","heartbeat_front_addr":"10.20.193.24:6815/1012255776","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":30,"last_clean_end":46,"up_from":50,"up_thru":50,"down_at":47,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":3968139738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":3968139738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":3968139738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":3968139738}]},"public_addr":"10.20.193.24:6801/3968139738","cluster_addr":"10.20.193.24:6802/3968139738","heartbeat_back_addr":"10.20.193.24:6804/3968139738","heartbeat_front_addr":"10.20.193.24:6803/3968139738","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.138188+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.488216+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.278612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:54.537485+0000","dead_epoch":47}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:18:32.134 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.3 is up 2026-02-17T15:18:32.135 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:18:32.289 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:18:32.289 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":51,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:18:29.220366+0000","last_up_change":"2026-02-17T15:18:28.217975+0000","last_in_change":"2026-02-17T15:17:45.881799+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":10,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-17T15:14:06.508581+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":36,"up_from":40,"up_thru":40,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":342414690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":342414690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":342414690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":342414690}]},"public_addr":"10.20.193.24:6809/342414690","cluster_addr":"10.20.193.24:6810/342414690","heartbeat_back_addr":"10.20.193.24:6812/342414690","heartbeat_front_addr":"10.20.193.24:6811/342414690","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":19,"up_from":23,"up_thru":47,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":2457068227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":2457068227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":2457068227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":2457068227}]},"public_addr":"10.20.193.24:6805/2457068227","cluster_addr":"10.20.193.24:6806/2457068227","heartbeat_back_addr":"10.20.193.24:6808/2457068227","heartbeat_front_addr":"10.20.193.24:6807/2457068227","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":1012255776}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":1012255776}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":1012255776}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":1012255776}]},"public_addr":"10.20.193.24:6813/1012255776","cluster_addr":"10.20.193.24:6814/1012255776","heartbeat_back_addr":"10.20.193.24:6816/1012255776","heartbeat_front_addr":"10.20.193.24:6815/1012255776","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":30,"last_clean_end":46,"up_from":50,"up_thru":50,"down_at":47,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":3968139738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":3968139738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":3968139738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":3968139738}]},"public_addr":"10.20.193.24:6801/3968139738","cluster_addr":"10.20.193.24:6802/3968139738","heartbeat_back_addr":"10.20.193.24:6804/3968139738","heartbeat_front_addr":"10.20.193.24:6803/3968139738","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.138188+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.488216+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.278612+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:54.537485+0000","dead_epoch":47}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:18:32.300 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json osd map unique_pool_0 test 2026-02-17T15:18:32.447 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:18:32.458 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":51,"pool":"unique_pool_0","pool_id":2,"objname":"test","raw_pgid":"2.40e8aab5","pgid":"2.0","up":[3,1,2],"up_primary":3,"acting":[3,1,2],"acting_primary":3} 2026-02-17T15:18:32.459 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-17T15:18:32.459 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T15:18:32.546 INFO:tasks.ceph.osd.2:Stopped 2026-02-17T15:18:32.546 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 --op list |grep '"oid":"test"') sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 "$object" rm-attr hinfo_key 2026-02-17T15:18:32.546 DEBUG:teuthology.orchestra.run.trial024:> bash -e -x -c 'object=$(sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 --op list |grep '"'"'"oid":"test"'"'"') 2026-02-17T15:18:32.546 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 "$object" rm-attr hinfo_key' 2026-02-17T15:18:32.550 INFO:teuthology.orchestra.run.trial024.stderr:+ '[' -z '' ']' 2026-02-17T15:18:32.551 INFO:teuthology.orchestra.run.trial024.stderr:+ return 2026-02-17T15:18:32.551 INFO:teuthology.orchestra.run.trial024.stderr:+ case $- in 2026-02-17T15:18:32.551 INFO:teuthology.orchestra.run.trial024.stderr:+ return 2026-02-17T15:18:32.551 INFO:teuthology.orchestra.run.trial024.stderr:++ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 --op list 2026-02-17T15:18:32.551 INFO:teuthology.orchestra.run.trial024.stderr:++ grep '"oid":"test"' 2026-02-17T15:18:35.112 INFO:teuthology.orchestra.run.trial024.stderr:+ object='["2.0s2",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":2,"max":0}]' 2026-02-17T15:18:35.112 INFO:teuthology.orchestra.run.trial024.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --pgid 2.0s2 '["2.0s2",{"oid":"test","key":"","snapid":-2,"hash":1088989877,"max":0,"pool":2,"namespace":"","shard_id":2,"max":0}]' rm-attr hinfo_key 2026-02-17T15:18:40.534 INFO:tasks.ceph.osd.2:Restarting daemon 2026-02-17T15:18:40.534 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2026-02-17T15:18:40.578 INFO:tasks.ceph.osd.2:Started 2026-02-17T15:18:40.578 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-17T15:18:40.636 INFO:teuthology.orchestra.run.trial024.stderr:no valid command found; 10 closest matches: 2026-02-17T15:18:40.636 INFO:teuthology.orchestra.run.trial024.stderr:0 2026-02-17T15:18:40.636 INFO:teuthology.orchestra.run.trial024.stderr:1 2026-02-17T15:18:40.637 INFO:teuthology.orchestra.run.trial024.stderr:2 2026-02-17T15:18:40.637 INFO:teuthology.orchestra.run.trial024.stderr:abort 2026-02-17T15:18:40.637 INFO:teuthology.orchestra.run.trial024.stderr:assert 2026-02-17T15:18:40.637 INFO:teuthology.orchestra.run.trial024.stderr:config diff 2026-02-17T15:18:40.637 INFO:teuthology.orchestra.run.trial024.stderr:config diff get 2026-02-17T15:18:40.637 INFO:teuthology.orchestra.run.trial024.stderr:config get 2026-02-17T15:18:40.637 INFO:teuthology.orchestra.run.trial024.stderr:config help [] 2026-02-17T15:18:40.637 INFO:teuthology.orchestra.run.trial024.stderr:config set ... 2026-02-17T15:18:40.637 INFO:teuthology.orchestra.run.trial024.stderr:admin_socket: invalid command 2026-02-17T15:18:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-17T15:18:40.639 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-17T15:18:41.838 INFO:tasks.ceph.osd.2.trial024.stderr:2026-02-17T15:18:41.836+0000 7ff23a6968c0 -1 Falling back to public interface 2026-02-17T15:18:45.640 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-17T15:18:45.700 INFO:teuthology.orchestra.run.trial024.stderr:no valid command found; 10 closest matches: 2026-02-17T15:18:45.701 INFO:teuthology.orchestra.run.trial024.stderr:0 2026-02-17T15:18:45.701 INFO:teuthology.orchestra.run.trial024.stderr:1 2026-02-17T15:18:45.701 INFO:teuthology.orchestra.run.trial024.stderr:2 2026-02-17T15:18:45.701 INFO:teuthology.orchestra.run.trial024.stderr:abort 2026-02-17T15:18:45.701 INFO:teuthology.orchestra.run.trial024.stderr:assert 2026-02-17T15:18:45.701 INFO:teuthology.orchestra.run.trial024.stderr:bluestore allocator dump block 2026-02-17T15:18:45.701 INFO:teuthology.orchestra.run.trial024.stderr:bluestore allocator fragmentation block 2026-02-17T15:18:45.701 INFO:teuthology.orchestra.run.trial024.stderr:bluestore allocator fragmentation histogram block [] [] 2026-02-17T15:18:45.701 INFO:teuthology.orchestra.run.trial024.stderr:bluestore allocator score block 2026-02-17T15:18:45.701 INFO:teuthology.orchestra.run.trial024.stderr:config diff 2026-02-17T15:18:45.702 INFO:teuthology.orchestra.run.trial024.stderr:admin_socket: invalid command 2026-02-17T15:18:45.702 DEBUG:teuthology.orchestra.run:got remote process result: 22 2026-02-17T15:18:45.703 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2026-02-17T15:18:45.994 INFO:tasks.ceph.osd.2.trial024.stderr:2026-02-17T15:18:45.992+0000 7ff23a6968c0 -1 osd.2 51 log_to_monitors true 2026-02-17T15:18:46.882 INFO:tasks.ceph.osd.2.trial024.stderr:2026-02-17T15:18:46.880+0000 7ff22e472640 -1 osd.2 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-17T15:18:50.704 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2026-02-17T15:18:50.768 INFO:teuthology.orchestra.run.trial024.stdout:{ 2026-02-17T15:18:50.768 INFO:teuthology.orchestra.run.trial024.stdout: "ops": [], 2026-02-17T15:18:50.768 INFO:teuthology.orchestra.run.trial024.stdout: "num_ops": 0 2026-02-17T15:18:50.768 INFO:teuthology.orchestra.run.trial024.stdout:} 2026-02-17T15:18:50.775 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for osd.2 to be up 2026-02-17T15:18:50.775 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:18:50.926 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:18:50.926 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":56,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:18:48.885017+0000","last_up_change":"2026-02-17T15:18:47.882890+0000","last_in_change":"2026-02-17T15:17:45.881799+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-17T15:14:06.508581+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":36,"up_from":40,"up_thru":40,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":342414690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":342414690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":342414690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":342414690}]},"public_addr":"10.20.193.24:6809/342414690","cluster_addr":"10.20.193.24:6810/342414690","heartbeat_back_addr":"10.20.193.24:6812/342414690","heartbeat_front_addr":"10.20.193.24:6811/342414690","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":19,"up_from":23,"up_thru":47,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":2457068227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":2457068227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":2457068227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":2457068227}]},"public_addr":"10.20.193.24:6805/2457068227","cluster_addr":"10.20.193.24:6806/2457068227","heartbeat_back_addr":"10.20.193.24:6808/2457068227","heartbeat_front_addr":"10.20.193.24:6807/2457068227","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":51,"up_from":55,"up_thru":37,"down_at":52,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":2608472267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":2608472267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":2608472267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":2608472267}]},"public_addr":"10.20.193.24:6813/2608472267","cluster_addr":"10.20.193.24:6814/2608472267","heartbeat_back_addr":"10.20.193.24:6816/2608472267","heartbeat_front_addr":"10.20.193.24:6815/2608472267","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":30,"last_clean_end":46,"up_from":50,"up_thru":55,"down_at":47,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":3968139738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":3968139738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":3968139738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":3968139738}]},"public_addr":"10.20.193.24:6801/3968139738","cluster_addr":"10.20.193.24:6802/3968139738","heartbeat_back_addr":"10.20.193.24:6804/3968139738","heartbeat_front_addr":"10.20.193.24:6803/3968139738","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.138188+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.488216+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.278612+0000","dead_epoch":52},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:54.537485+0000","dead_epoch":47}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:18:50.939 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:osd.2 is up 2026-02-17T15:18:50.939 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2026-02-17T15:18:51.101 INFO:tasks.ceph.mon.a.trial024.stderr:2026-02-17T15:18:51.096+0000 7fc1ce14a640 -1 mon.a@0(leader).osd e56 definitely_dead 0 2026-02-17T15:18:51.983 INFO:teuthology.orchestra.run.trial024.stderr:marked out osd.2. 2026-02-17T15:18:51.998 DEBUG:tasks.ec_inconsistent_hinfo:waiting for pg 2.0 backfill from osd.2 to osd.0 complete 2026-02-17T15:18:57.000 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:18:57.001 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:18:57.001 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:18:57.002 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:18:57.089 INFO:teuthology.orchestra.run.trial024.stdout:214748364807 2026-02-17T15:18:57.090 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:18:57.091 INFO:teuthology.orchestra.run.trial024.stdout:236223201284 2026-02-17T15:18:57.092 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:18:57.093 INFO:teuthology.orchestra.run.trial024.stdout:98784247886 2026-02-17T15:18:57.093 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:18:57.094 INFO:teuthology.orchestra.run.trial024.stdout:171798691861 2026-02-17T15:18:57.095 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:18:57.257 INFO:teuthology.orchestra.run.trial024.stdout:214748364806 2026-02-17T15:18:57.272 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364807 got 214748364806 for osd.3 2026-02-17T15:18:57.282 INFO:teuthology.orchestra.run.trial024.stdout:98784247885 2026-02-17T15:18:57.285 INFO:teuthology.orchestra.run.trial024.stdout:0 2026-02-17T15:18:57.289 INFO:teuthology.orchestra.run.trial024.stdout:171798691859 2026-02-17T15:18:57.295 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247886 got 98784247885 for osd.1 2026-02-17T15:18:57.297 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201284 got 0 for osd.2 2026-02-17T15:18:57.301 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691861 got 171798691859 for osd.0 2026-02-17T15:18:58.273 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:18:58.296 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:18:58.298 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:18:58.302 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:18:58.433 INFO:teuthology.orchestra.run.trial024.stdout:214748364808 2026-02-17T15:18:58.446 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364807 got 214748364808 for osd.3 2026-02-17T15:18:58.446 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:58.469 INFO:teuthology.orchestra.run.trial024.stdout:236223201284 2026-02-17T15:18:58.481 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201284 got 236223201284 for osd.2 2026-02-17T15:18:58.482 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:58.489 INFO:teuthology.orchestra.run.trial024.stdout:98784247886 2026-02-17T15:18:58.492 INFO:teuthology.orchestra.run.trial024.stdout:171798691861 2026-02-17T15:18:58.501 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247886 got 98784247886 for osd.1 2026-02-17T15:18:58.501 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:58.504 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691861 got 171798691861 for osd.0 2026-02-17T15:18:58.504 DEBUG:teuthology.parallel:result is None 2026-02-17T15:18:58.505 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:18:58.687 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:18:58.688 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:18:58.698 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2026-02-17T15:18:57.501075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":69238,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74242,"num_bytes_recovered":608494003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":851164,"kb_used_data":579736,"kb_used_omap":14,"kb_used_meta":271345,"kb_avail":282264356,"statfs":{"total":289910292480,"available":289038700544,"internally_reserved":0,"allocated":593649664,"data_stored":593384850,"data_compressed":10310,"data_compressed_allocated":57344,"data_compressed_original":688128,"omap_allocated":14891,"internal_metadata":277857749},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-71529,"num_objects_misplaced":69238,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2292,"num_bytes_recovered":18776064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000790"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":220197,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:18:57.223259+0000","last_change":"2026-02-17T15:18:54.281346+0000","last_active":"2026-02-17T15:18:57.223259+0000","last_peered":"2026-02-17T15:18:57.223259+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:18:57.223259+0000","last_undegraded":"2026-02-17T15:18:57.223259+0000","last_fullsized":"2026-02-17T15:18:57.223259+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":69238,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74240,"num_bytes_recovered":608166323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989512+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989512+0000","last_peered":"2026-02-17T15:18:53.989512+0000","last_clean":"2026-02-17T15:18:53.989512+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989512+0000","last_undegraded":"2026-02-17T15:18:53.989512+0000","last_fullsized":"2026-02-17T15:18:53.989512+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":69238,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74240,"num_bytes_recovered":608166323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":887816192,"data_stored":887840768,"data_compressed":16028,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247886,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":293490688,"data_stored":293508742,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364808,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326560,"kb_used_data":286596,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045280,"statfs":{"total":96636764160,"available":96302366720,"internally_reserved":0,"allocated":293474304,"data_stored":293185158,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201284,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691861,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53584,"kb_used_data":6528,"kb_used_omap":4,"kb_used_meta":47035,"kb_avail":94318256,"statfs":{"total":96636764160,"available":96581894144,"internally_reserved":0,"allocated":6684672,"data_stored":6690950,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":4404,"internal_metadata":48164556},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8871936,"data_stored":8871936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:18:58.699 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 220197, 'reported_epoch': 59, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:18:57.223259+0000', 'last_change': '2026-02-17T15:18:54.281346+0000', 'last_active': '2026-02-17T15:18:57.223259+0000', 'last_peered': '2026-02-17T15:18:57.223259+0000', 'last_clean': '2026-02-17T15:18:48.890765+0000', 'last_became_active': '2026-02-17T15:18:54.280577+0000', 'last_became_peered': '2026-02-17T15:18:54.280577+0000', 'last_unstale': '2026-02-17T15:18:57.223259+0000', 'last_undegraded': '2026-02-17T15:18:57.223259+0000', 'last_fullsized': '2026-02-17T15:18:57.223259+0000', 'mapping_epoch': 58, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 69238, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 74240, 'num_bytes_recovered': 608166323, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:18:58.699 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:19:03.700 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:19:03.701 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:19:03.702 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:19:03.702 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:19:03.788 INFO:teuthology.orchestra.run.trial024.stdout:98784247888 2026-02-17T15:19:03.789 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:03.790 INFO:teuthology.orchestra.run.trial024.stdout:236223201286 2026-02-17T15:19:03.791 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:03.792 INFO:teuthology.orchestra.run.trial024.stdout:171798691863 2026-02-17T15:19:03.792 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:03.794 INFO:teuthology.orchestra.run.trial024.stdout:214748364810 2026-02-17T15:19:03.795 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:03.963 INFO:teuthology.orchestra.run.trial024.stdout:98784247887 2026-02-17T15:19:03.977 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247888 got 98784247887 for osd.1 2026-02-17T15:19:03.988 INFO:teuthology.orchestra.run.trial024.stdout:214748364809 2026-02-17T15:19:03.993 INFO:teuthology.orchestra.run.trial024.stdout:236223201285 2026-02-17T15:19:04.001 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364810 got 214748364809 for osd.3 2026-02-17T15:19:04.005 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201286 got 236223201285 for osd.2 2026-02-17T15:19:04.019 INFO:teuthology.orchestra.run.trial024.stdout:171798691862 2026-02-17T15:19:04.031 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691863 got 171798691862 for osd.0 2026-02-17T15:19:04.978 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:05.002 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:05.006 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:05.032 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:05.132 INFO:teuthology.orchestra.run.trial024.stdout:98784247887 2026-02-17T15:19:05.145 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247888 got 98784247887 for osd.1 2026-02-17T15:19:05.185 INFO:teuthology.orchestra.run.trial024.stdout:214748364809 2026-02-17T15:19:05.191 INFO:teuthology.orchestra.run.trial024.stdout:236223201285 2026-02-17T15:19:05.198 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364810 got 214748364809 for osd.3 2026-02-17T15:19:05.205 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201286 got 236223201285 for osd.2 2026-02-17T15:19:05.216 INFO:teuthology.orchestra.run.trial024.stdout:171798691862 2026-02-17T15:19:05.227 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691863 got 171798691862 for osd.0 2026-02-17T15:19:06.146 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:06.199 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:06.206 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:06.228 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:06.307 INFO:teuthology.orchestra.run.trial024.stdout:98784247889 2026-02-17T15:19:06.322 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247888 got 98784247889 for osd.1 2026-02-17T15:19:06.322 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:06.353 INFO:teuthology.orchestra.run.trial024.stdout:214748364810 2026-02-17T15:19:06.366 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364810 got 214748364810 for osd.3 2026-02-17T15:19:06.366 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:06.396 INFO:teuthology.orchestra.run.trial024.stdout:236223201286 2026-02-17T15:19:06.400 INFO:teuthology.orchestra.run.trial024.stdout:171798691863 2026-02-17T15:19:06.409 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201286 got 236223201286 for osd.2 2026-02-17T15:19:06.409 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:06.412 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691863 got 171798691863 for osd.0 2026-02-17T15:19:06.412 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:06.413 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:19:06.613 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:19:06.614 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:19:06.625 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":211,"stamp":"2026-02-17T15:19:05.502007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":64017,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79463,"num_bytes_recovered":651264435,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":905740,"kb_used_data":602376,"kb_used_omap":14,"kb_used_meta":303281,"kb_avail":282209780,"statfs":{"total":289910292480,"available":288982814720,"internally_reserved":0,"allocated":616833024,"data_stored":616568210,"data_compressed":10310,"data_compressed_allocated":57344,"data_compressed_original":688128,"omap_allocated":14891,"internal_metadata":310560213},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":64017,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7513,"num_bytes_recovered":61546496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.513229"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":230639,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:19:03.781677+0000","last_change":"2026-02-17T15:18:54.281346+0000","last_active":"2026-02-17T15:19:03.781677+0000","last_peered":"2026-02-17T15:19:03.781677+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:19:03.781677+0000","last_undegraded":"2026-02-17T15:19:03.781677+0000","last_fullsized":"2026-02-17T15:19:03.781677+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":64017,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79461,"num_bytes_recovered":650936755,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989512+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989512+0000","last_peered":"2026-02-17T15:18:53.989512+0000","last_clean":"2026-02-17T15:18:53.989512+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989512+0000","last_undegraded":"2026-02-17T15:18:53.989512+0000","last_fullsized":"2026-02-17T15:18:53.989512+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":64017,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79461,"num_bytes_recovered":650936755,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909705216,"data_stored":909729792,"data_compressed":16028,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247889,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":293490688,"data_stored":293508742,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364810,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326560,"kb_used_data":286596,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045280,"statfs":{"total":96636764160,"available":96302366720,"internally_reserved":0,"allocated":293474304,"data_stored":293185158,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201286,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691863,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":108160,"kb_used_data":29168,"kb_used_omap":4,"kb_used_meta":78971,"kb_avail":94263680,"statfs":{"total":96636764160,"available":96526008320,"internally_reserved":0,"allocated":29868032,"data_stored":29874310,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":4404,"internal_metadata":80867020},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":30760960,"data_stored":30760960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:19:06.626 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 230639, 'reported_epoch': 59, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:19:03.781677+0000', 'last_change': '2026-02-17T15:18:54.281346+0000', 'last_active': '2026-02-17T15:19:03.781677+0000', 'last_peered': '2026-02-17T15:19:03.781677+0000', 'last_clean': '2026-02-17T15:18:48.890765+0000', 'last_became_active': '2026-02-17T15:18:54.280577+0000', 'last_became_peered': '2026-02-17T15:18:54.280577+0000', 'last_unstale': '2026-02-17T15:19:03.781677+0000', 'last_undegraded': '2026-02-17T15:19:03.781677+0000', 'last_fullsized': '2026-02-17T15:19:03.781677+0000', 'mapping_epoch': 58, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 64017, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 79461, 'num_bytes_recovered': 650936755, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:19:06.626 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:19:11.627 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:19:11.628 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:19:11.629 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:19:11.629 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:19:11.717 INFO:teuthology.orchestra.run.trial024.stdout:171798691866 2026-02-17T15:19:11.717 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:11.718 INFO:teuthology.orchestra.run.trial024.stdout:236223201289 2026-02-17T15:19:11.719 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:11.720 INFO:teuthology.orchestra.run.trial024.stdout:98784247891 2026-02-17T15:19:11.721 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:11.722 INFO:teuthology.orchestra.run.trial024.stdout:214748364812 2026-02-17T15:19:11.723 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:11.873 INFO:teuthology.orchestra.run.trial024.stdout:236223201287 2026-02-17T15:19:11.886 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201289 got 236223201287 for osd.2 2026-02-17T15:19:11.889 INFO:teuthology.orchestra.run.trial024.stdout:171798691865 2026-02-17T15:19:11.901 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691866 got 171798691865 for osd.0 2026-02-17T15:19:11.925 INFO:teuthology.orchestra.run.trial024.stdout:214748364811 2026-02-17T15:19:11.928 INFO:teuthology.orchestra.run.trial024.stdout:98784247890 2026-02-17T15:19:11.937 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364812 got 214748364811 for osd.3 2026-02-17T15:19:11.941 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247891 got 98784247890 for osd.1 2026-02-17T15:19:12.887 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:12.902 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:12.938 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:12.942 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:13.038 INFO:teuthology.orchestra.run.trial024.stdout:236223201287 2026-02-17T15:19:13.051 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201289 got 236223201287 for osd.2 2026-02-17T15:19:13.085 INFO:teuthology.orchestra.run.trial024.stdout:171798691865 2026-02-17T15:19:13.097 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691866 got 171798691865 for osd.0 2026-02-17T15:19:13.140 INFO:teuthology.orchestra.run.trial024.stdout:214748364811 2026-02-17T15:19:13.149 INFO:teuthology.orchestra.run.trial024.stdout:98784247890 2026-02-17T15:19:13.154 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364812 got 214748364811 for osd.3 2026-02-17T15:19:13.163 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247891 got 98784247890 for osd.1 2026-02-17T15:19:14.052 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:14.098 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:14.155 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:14.163 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:14.204 INFO:teuthology.orchestra.run.trial024.stdout:236223201289 2026-02-17T15:19:14.219 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201289 got 236223201289 for osd.2 2026-02-17T15:19:14.220 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:14.255 INFO:teuthology.orchestra.run.trial024.stdout:171798691866 2026-02-17T15:19:14.268 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691866 got 171798691866 for osd.0 2026-02-17T15:19:14.268 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:14.307 INFO:teuthology.orchestra.run.trial024.stdout:214748364813 2026-02-17T15:19:14.320 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364812 got 214748364813 for osd.3 2026-02-17T15:19:14.320 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:14.350 INFO:teuthology.orchestra.run.trial024.stdout:98784247891 2026-02-17T15:19:14.362 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247891 got 98784247891 for osd.1 2026-02-17T15:19:14.362 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:14.362 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:19:14.552 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:19:14.552 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:19:14.562 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2026-02-17T15:19:13.502906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":57290,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86190,"num_bytes_recovered":706372156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":935592,"kb_used_data":628068,"kb_used_omap":14,"kb_used_meta":307441,"kb_avail":282179928,"statfs":{"total":289910292480,"available":288952246272,"internally_reserved":0,"allocated":643141632,"data_stored":642880914,"data_compressed":13762,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14891,"internal_metadata":314820053},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-11948,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11948,"num_bytes_recovered":97878153,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001341"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":244093,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:19:12.224878+0000","last_change":"2026-02-17T15:18:54.281346+0000","last_active":"2026-02-17T15:19:12.224878+0000","last_peered":"2026-02-17T15:19:12.224878+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:19:12.224878+0000","last_undegraded":"2026-02-17T15:19:12.224878+0000","last_fullsized":"2026-02-17T15:19:12.224878+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":57290,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86188,"num_bytes_recovered":706044476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989512+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989512+0000","last_peered":"2026-02-17T15:18:53.989512+0000","last_clean":"2026-02-17T15:18:53.989512+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989512+0000","last_undegraded":"2026-02-17T15:18:53.989512+0000","last_fullsized":"2026-02-17T15:18:53.989512+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":57290,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86188,"num_bytes_recovered":706044476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":935485440,"data_stored":935514112,"data_compressed":19480,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247891,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":293490688,"data_stored":293508742,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364813,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326560,"kb_used_data":286596,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045280,"statfs":{"total":96636764160,"available":96302366720,"internally_reserved":0,"allocated":293474304,"data_stored":293185158,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201289,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691866,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138012,"kb_used_data":54860,"kb_used_omap":4,"kb_used_meta":83131,"kb_avail":94233828,"statfs":{"total":96636764160,"available":96495439872,"internally_reserved":0,"allocated":56176640,"data_stored":56187014,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":85126860},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56541184,"data_stored":56545280,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:19:14.563 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 244093, 'reported_epoch': 59, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:19:12.224878+0000', 'last_change': '2026-02-17T15:18:54.281346+0000', 'last_active': '2026-02-17T15:19:12.224878+0000', 'last_peered': '2026-02-17T15:19:12.224878+0000', 'last_clean': '2026-02-17T15:18:48.890765+0000', 'last_became_active': '2026-02-17T15:18:54.280577+0000', 'last_became_peered': '2026-02-17T15:18:54.280577+0000', 'last_unstale': '2026-02-17T15:19:12.224878+0000', 'last_undegraded': '2026-02-17T15:19:12.224878+0000', 'last_fullsized': '2026-02-17T15:19:12.224878+0000', 'mapping_epoch': 58, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 57290, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 86188, 'num_bytes_recovered': 706044476, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:19:14.563 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:19:19.565 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:19:19.566 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:19:19.566 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:19:19.567 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:19:19.653 INFO:teuthology.orchestra.run.trial024.stdout:236223201291 2026-02-17T15:19:19.653 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:19.655 INFO:teuthology.orchestra.run.trial024.stdout:98784247893 2026-02-17T15:19:19.655 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:19.657 INFO:teuthology.orchestra.run.trial024.stdout:171798691868 2026-02-17T15:19:19.657 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:19.661 INFO:teuthology.orchestra.run.trial024.stdout:214748364815 2026-02-17T15:19:19.661 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:19.804 INFO:teuthology.orchestra.run.trial024.stdout:236223201290 2026-02-17T15:19:19.812 INFO:teuthology.orchestra.run.trial024.stdout:98784247892 2026-02-17T15:19:19.817 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201291 got 236223201290 for osd.2 2026-02-17T15:19:19.827 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247893 got 98784247892 for osd.1 2026-02-17T15:19:19.851 INFO:teuthology.orchestra.run.trial024.stdout:171798691867 2026-02-17T15:19:19.862 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691868 got 171798691867 for osd.0 2026-02-17T15:19:19.866 INFO:teuthology.orchestra.run.trial024.stdout:214748364814 2026-02-17T15:19:19.878 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364815 got 214748364814 for osd.3 2026-02-17T15:19:20.818 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:20.828 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:20.863 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:20.879 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:20.969 INFO:teuthology.orchestra.run.trial024.stdout:236223201290 2026-02-17T15:19:20.983 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201291 got 236223201290 for osd.2 2026-02-17T15:19:21.028 INFO:teuthology.orchestra.run.trial024.stdout:171798691867 2026-02-17T15:19:21.029 INFO:teuthology.orchestra.run.trial024.stdout:98784247892 2026-02-17T15:19:21.042 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247893 got 98784247892 for osd.1 2026-02-17T15:19:21.043 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691868 got 171798691867 for osd.0 2026-02-17T15:19:21.071 INFO:teuthology.orchestra.run.trial024.stdout:214748364814 2026-02-17T15:19:21.083 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364815 got 214748364814 for osd.3 2026-02-17T15:19:21.984 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:22.043 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:22.044 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:22.084 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:22.141 INFO:teuthology.orchestra.run.trial024.stdout:236223201291 2026-02-17T15:19:22.154 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201291 got 236223201291 for osd.2 2026-02-17T15:19:22.155 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:22.195 INFO:teuthology.orchestra.run.trial024.stdout:98784247894 2026-02-17T15:19:22.195 INFO:teuthology.orchestra.run.trial024.stdout:171798691869 2026-02-17T15:19:22.207 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691868 got 171798691869 for osd.0 2026-02-17T15:19:22.208 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:22.209 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247893 got 98784247894 for osd.1 2026-02-17T15:19:22.209 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:22.253 INFO:teuthology.orchestra.run.trial024.stdout:214748364815 2026-02-17T15:19:22.265 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364815 got 214748364815 for osd.3 2026-02-17T15:19:22.265 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:22.265 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:19:22.415 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:19:22.416 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:19:22.426 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2026-02-17T15:19:21.503886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":51235,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92245,"num_bytes_recovered":755974716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":992648,"kb_used_data":657348,"kb_used_omap":14,"kb_used_meta":335217,"kb_avail":282122872,"statfs":{"total":289910292480,"available":288893820928,"internally_reserved":0,"allocated":673124352,"data_stored":672863634,"data_compressed":13762,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14891,"internal_metadata":343262677},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-10001,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10001,"num_bytes_recovered":81928329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001425"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":256203,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:19:19.648447+0000","last_change":"2026-02-17T15:18:54.281346+0000","last_active":"2026-02-17T15:19:19.648447+0000","last_peered":"2026-02-17T15:19:19.648447+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:19:19.648447+0000","last_undegraded":"2026-02-17T15:19:19.648447+0000","last_fullsized":"2026-02-17T15:19:19.648447+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":51235,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92243,"num_bytes_recovered":755647036,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989512+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989512+0000","last_peered":"2026-02-17T15:18:53.989512+0000","last_clean":"2026-02-17T15:18:53.989512+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989512+0000","last_undegraded":"2026-02-17T15:18:53.989512+0000","last_fullsized":"2026-02-17T15:18:53.989512+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":51235,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92243,"num_bytes_recovered":755647036,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":965136384,"data_stored":965165056,"data_compressed":19480,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247894,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":293490688,"data_stored":293508742,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364815,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326560,"kb_used_data":286596,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045280,"statfs":{"total":96636764160,"available":96302366720,"internally_reserved":0,"allocated":293474304,"data_stored":293185158,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201291,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":195068,"kb_used_data":84140,"kb_used_omap":4,"kb_used_meta":110907,"kb_avail":94176772,"statfs":{"total":96636764160,"available":96437014528,"internally_reserved":0,"allocated":86159360,"data_stored":86169734,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":113569484},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86192128,"data_stored":86196224,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:19:22.427 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 256203, 'reported_epoch': 59, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:19:19.648447+0000', 'last_change': '2026-02-17T15:18:54.281346+0000', 'last_active': '2026-02-17T15:19:19.648447+0000', 'last_peered': '2026-02-17T15:19:19.648447+0000', 'last_clean': '2026-02-17T15:18:48.890765+0000', 'last_became_active': '2026-02-17T15:18:54.280577+0000', 'last_became_peered': '2026-02-17T15:18:54.280577+0000', 'last_unstale': '2026-02-17T15:19:19.648447+0000', 'last_undegraded': '2026-02-17T15:19:19.648447+0000', 'last_fullsized': '2026-02-17T15:19:19.648447+0000', 'mapping_epoch': 58, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 51235, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 92243, 'num_bytes_recovered': 755647036, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:19:22.428 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:19:27.428 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:19:27.429 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:19:27.430 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:19:27.430 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:19:27.519 INFO:teuthology.orchestra.run.trial024.stdout:98784247896 2026-02-17T15:19:27.519 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:27.521 INFO:teuthology.orchestra.run.trial024.stdout:171798691871 2026-02-17T15:19:27.521 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:27.522 INFO:teuthology.orchestra.run.trial024.stdout:214748364818 2026-02-17T15:19:27.523 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:27.529 INFO:teuthology.orchestra.run.trial024.stdout:236223201294 2026-02-17T15:19:27.530 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:27.686 INFO:teuthology.orchestra.run.trial024.stdout:98784247895 2026-02-17T15:19:27.700 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247896 got 98784247895 for osd.1 2026-02-17T15:19:27.738 INFO:teuthology.orchestra.run.trial024.stdout:171798691870 2026-02-17T15:19:27.751 INFO:teuthology.orchestra.run.trial024.stdout:214748364817 2026-02-17T15:19:27.752 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691871 got 171798691870 for osd.0 2026-02-17T15:19:27.753 INFO:teuthology.orchestra.run.trial024.stdout:236223201293 2026-02-17T15:19:27.763 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364818 got 214748364817 for osd.3 2026-02-17T15:19:27.765 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201294 got 236223201293 for osd.2 2026-02-17T15:19:28.700 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:28.753 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:28.764 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:28.766 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:28.854 INFO:teuthology.orchestra.run.trial024.stdout:98784247895 2026-02-17T15:19:28.867 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247896 got 98784247895 for osd.1 2026-02-17T15:19:28.914 INFO:teuthology.orchestra.run.trial024.stdout:171798691870 2026-02-17T15:19:28.929 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691871 got 171798691870 for osd.0 2026-02-17T15:19:28.959 INFO:teuthology.orchestra.run.trial024.stdout:214748364817 2026-02-17T15:19:28.964 INFO:teuthology.orchestra.run.trial024.stdout:236223201293 2026-02-17T15:19:28.972 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364818 got 214748364817 for osd.3 2026-02-17T15:19:28.976 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201294 got 236223201293 for osd.2 2026-02-17T15:19:29.868 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:29.930 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:29.972 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:29.977 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:30.015 INFO:teuthology.orchestra.run.trial024.stdout:98784247896 2026-02-17T15:19:30.028 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247896 got 98784247896 for osd.1 2026-02-17T15:19:30.028 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:30.092 INFO:teuthology.orchestra.run.trial024.stdout:171798691871 2026-02-17T15:19:30.106 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691871 got 171798691871 for osd.0 2026-02-17T15:19:30.106 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:30.126 INFO:teuthology.orchestra.run.trial024.stdout:214748364818 2026-02-17T15:19:30.137 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364818 got 214748364818 for osd.3 2026-02-17T15:19:30.138 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:30.146 INFO:teuthology.orchestra.run.trial024.stdout:236223201294 2026-02-17T15:19:30.159 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201294 got 236223201294 for osd.2 2026-02-17T15:19:30.159 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:30.159 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:19:30.357 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:19:30.357 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:19:30.370 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2026-02-17T15:19:29.504890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":45060,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98420,"num_bytes_recovered":806560316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1063136,"kb_used_data":679260,"kb_used_omap":14,"kb_used_meta":383793,"kb_avail":282052384,"statfs":{"total":289910292480,"available":288821641216,"internally_reserved":0,"allocated":695562240,"data_stored":695301522,"data_compressed":13762,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14891,"internal_metadata":393004501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8203,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8203,"num_bytes_recovered":67198976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001504"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":268553,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:19:27.508582+0000","last_change":"2026-02-17T15:18:54.281346+0000","last_active":"2026-02-17T15:19:27.508582+0000","last_peered":"2026-02-17T15:19:27.508582+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:19:27.508582+0000","last_undegraded":"2026-02-17T15:19:27.508582+0000","last_fullsized":"2026-02-17T15:19:27.508582+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":45060,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98418,"num_bytes_recovered":806232636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989512+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989512+0000","last_peered":"2026-02-17T15:18:53.989512+0000","last_clean":"2026-02-17T15:18:53.989512+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989512+0000","last_undegraded":"2026-02-17T15:18:53.989512+0000","last_fullsized":"2026-02-17T15:18:53.989512+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":45060,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98418,"num_bytes_recovered":806232636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":987361280,"data_stored":987389952,"data_compressed":19480,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247896,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":293490688,"data_stored":293508742,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364818,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326560,"kb_used_data":286596,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045280,"statfs":{"total":96636764160,"available":96302366720,"internally_reserved":0,"allocated":293474304,"data_stored":293185158,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201294,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691871,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265556,"kb_used_data":106052,"kb_used_omap":4,"kb_used_meta":159483,"kb_avail":94106284,"statfs":{"total":96636764160,"available":96364834816,"internally_reserved":0,"allocated":108597248,"data_stored":108607622,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":163311308},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":108417024,"data_stored":108421120,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:19:30.371 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 268553, 'reported_epoch': 59, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:19:27.508582+0000', 'last_change': '2026-02-17T15:18:54.281346+0000', 'last_active': '2026-02-17T15:19:27.508582+0000', 'last_peered': '2026-02-17T15:19:27.508582+0000', 'last_clean': '2026-02-17T15:18:48.890765+0000', 'last_became_active': '2026-02-17T15:18:54.280577+0000', 'last_became_peered': '2026-02-17T15:18:54.280577+0000', 'last_unstale': '2026-02-17T15:19:27.508582+0000', 'last_undegraded': '2026-02-17T15:19:27.508582+0000', 'last_fullsized': '2026-02-17T15:19:27.508582+0000', 'mapping_epoch': 58, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 45060, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 98418, 'num_bytes_recovered': 806232636, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:19:30.371 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:19:35.373 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:19:35.374 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:19:35.374 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:19:35.375 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:19:35.464 INFO:teuthology.orchestra.run.trial024.stdout:171798691873 2026-02-17T15:19:35.465 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:35.468 INFO:teuthology.orchestra.run.trial024.stdout:98784247899 2026-02-17T15:19:35.468 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:35.469 INFO:teuthology.orchestra.run.trial024.stdout:214748364820 2026-02-17T15:19:35.470 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:35.471 INFO:teuthology.orchestra.run.trial024.stdout:236223201296 2026-02-17T15:19:35.471 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:35.617 INFO:teuthology.orchestra.run.trial024.stdout:171798691873 2026-02-17T15:19:35.633 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691873 got 171798691873 for osd.0 2026-02-17T15:19:35.633 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:35.665 INFO:teuthology.orchestra.run.trial024.stdout:236223201296 2026-02-17T15:19:35.667 INFO:teuthology.orchestra.run.trial024.stdout:98784247899 2026-02-17T15:19:35.677 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201296 got 236223201296 for osd.2 2026-02-17T15:19:35.678 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:35.679 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247899 got 98784247899 for osd.1 2026-02-17T15:19:35.679 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:35.689 INFO:teuthology.orchestra.run.trial024.stdout:214748364820 2026-02-17T15:19:35.704 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364820 got 214748364820 for osd.3 2026-02-17T15:19:35.704 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:35.705 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:19:35.887 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:19:35.888 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:19:35.897 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":226,"stamp":"2026-02-17T15:19:35.505645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":38764,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104716,"num_bytes_recovered":858137148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1124484,"kb_used_data":704512,"kb_used_omap":14,"kb_used_meta":419889,"kb_avail":281991036,"statfs":{"total":289910292480,"available":288758820864,"internally_reserved":0,"allocated":721420288,"data_stored":721159570,"data_compressed":13762,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14891,"internal_metadata":429966805},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-10415,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10415,"num_bytes_recovered":85319680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001479"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":281145,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:19:35.453010+0000","last_change":"2026-02-17T15:18:54.281346+0000","last_active":"2026-02-17T15:19:35.453010+0000","last_peered":"2026-02-17T15:19:35.453010+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:19:35.453010+0000","last_undegraded":"2026-02-17T15:19:35.453010+0000","last_fullsized":"2026-02-17T15:19:35.453010+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":38764,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104714,"num_bytes_recovered":857809468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989512+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989512+0000","last_peered":"2026-02-17T15:18:53.989512+0000","last_clean":"2026-02-17T15:18:53.989512+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989512+0000","last_undegraded":"2026-02-17T15:18:53.989512+0000","last_fullsized":"2026-02-17T15:18:53.989512+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":38764,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104714,"num_bytes_recovered":857809468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1013141504,"data_stored":1013170176,"data_compressed":19480,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247899,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":293490688,"data_stored":293508742,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364820,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326560,"kb_used_data":286596,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045280,"statfs":{"total":96636764160,"available":96302366720,"internally_reserved":0,"allocated":293474304,"data_stored":293185158,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201296,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326904,"kb_used_data":131304,"kb_used_omap":4,"kb_used_meta":195579,"kb_avail":94044936,"statfs":{"total":96636764160,"available":96302014464,"internally_reserved":0,"allocated":134455296,"data_stored":134465670,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":200273612},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134197248,"data_stored":134201344,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:19:35.899 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 281145, 'reported_epoch': 59, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:19:35.453010+0000', 'last_change': '2026-02-17T15:18:54.281346+0000', 'last_active': '2026-02-17T15:19:35.453010+0000', 'last_peered': '2026-02-17T15:19:35.453010+0000', 'last_clean': '2026-02-17T15:18:48.890765+0000', 'last_became_active': '2026-02-17T15:18:54.280577+0000', 'last_became_peered': '2026-02-17T15:18:54.280577+0000', 'last_unstale': '2026-02-17T15:19:35.453010+0000', 'last_undegraded': '2026-02-17T15:19:35.453010+0000', 'last_fullsized': '2026-02-17T15:19:35.453010+0000', 'mapping_epoch': 58, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 38764, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 104714, 'num_bytes_recovered': 857809468, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:19:35.899 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:19:40.900 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:19:40.901 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:19:40.901 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:19:40.902 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:19:40.987 INFO:teuthology.orchestra.run.trial024.stdout:214748364822 2026-02-17T15:19:40.987 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:40.989 INFO:teuthology.orchestra.run.trial024.stdout:98784247901 2026-02-17T15:19:40.989 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:40.990 INFO:teuthology.orchestra.run.trial024.stdout:236223201298 2026-02-17T15:19:40.991 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:40.992 INFO:teuthology.orchestra.run.trial024.stdout:171798691876 2026-02-17T15:19:40.992 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:41.140 INFO:teuthology.orchestra.run.trial024.stdout:214748364821 2026-02-17T15:19:41.152 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364822 got 214748364821 for osd.3 2026-02-17T15:19:41.179 INFO:teuthology.orchestra.run.trial024.stdout:98784247899 2026-02-17T15:19:41.191 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247901 got 98784247899 for osd.1 2026-02-17T15:19:41.199 INFO:teuthology.orchestra.run.trial024.stdout:171798691874 2026-02-17T15:19:41.200 INFO:teuthology.orchestra.run.trial024.stdout:236223201297 2026-02-17T15:19:41.212 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691876 got 171798691874 for osd.0 2026-02-17T15:19:41.213 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201298 got 236223201297 for osd.2 2026-02-17T15:19:42.153 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:42.192 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:42.213 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:42.214 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:42.303 INFO:teuthology.orchestra.run.trial024.stdout:214748364822 2026-02-17T15:19:42.316 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364822 got 214748364822 for osd.3 2026-02-17T15:19:42.316 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:42.347 INFO:teuthology.orchestra.run.trial024.stdout:98784247901 2026-02-17T15:19:42.359 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247901 got 98784247901 for osd.1 2026-02-17T15:19:42.360 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:42.397 INFO:teuthology.orchestra.run.trial024.stdout:236223201298 2026-02-17T15:19:42.397 INFO:teuthology.orchestra.run.trial024.stdout:171798691876 2026-02-17T15:19:42.409 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691876 got 171798691876 for osd.0 2026-02-17T15:19:42.410 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:42.410 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201298 got 236223201298 for osd.2 2026-02-17T15:19:42.411 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:42.411 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:19:42.592 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:19:42.593 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:19:42.603 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2026-02-17T15:19:41.506383+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34299,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109181,"num_bytes_recovered":894714428,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1176892,"kb_used_data":720824,"kb_used_omap":14,"kb_used_meta":455985,"kb_avail":281938628,"statfs":{"total":289910292480,"available":288705155072,"internally_reserved":0,"allocated":738123776,"data_stored":737863058,"data_compressed":13762,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14891,"internal_metadata":466929109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-10761,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10761,"num_bytes_recovered":88154112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":290075,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:19:40.974826+0000","last_change":"2026-02-17T15:18:54.281346+0000","last_active":"2026-02-17T15:19:40.974826+0000","last_peered":"2026-02-17T15:19:40.974826+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:19:40.974826+0000","last_undegraded":"2026-02-17T15:19:40.974826+0000","last_fullsized":"2026-02-17T15:19:40.974826+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34299,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109179,"num_bytes_recovered":894386748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989512+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989512+0000","last_peered":"2026-02-17T15:18:53.989512+0000","last_clean":"2026-02-17T15:18:53.989512+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989512+0000","last_undegraded":"2026-02-17T15:18:53.989512+0000","last_fullsized":"2026-02-17T15:18:53.989512+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34299,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109179,"num_bytes_recovered":894386748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1031446528,"data_stored":1031475200,"data_compressed":19480,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247901,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":293490688,"data_stored":293508742,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364822,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326560,"kb_used_data":286596,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045280,"statfs":{"total":96636764160,"available":96302366720,"internally_reserved":0,"allocated":293474304,"data_stored":293185158,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201298,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691876,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":379312,"kb_used_data":147616,"kb_used_omap":4,"kb_used_meta":231675,"kb_avail":93992528,"statfs":{"total":96636764160,"available":96248348672,"internally_reserved":0,"allocated":151158784,"data_stored":151169158,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":237235916},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":152502272,"data_stored":152506368,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:19:42.605 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 290075, 'reported_epoch': 59, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:19:40.974826+0000', 'last_change': '2026-02-17T15:18:54.281346+0000', 'last_active': '2026-02-17T15:19:40.974826+0000', 'last_peered': '2026-02-17T15:19:40.974826+0000', 'last_clean': '2026-02-17T15:18:48.890765+0000', 'last_became_active': '2026-02-17T15:18:54.280577+0000', 'last_became_peered': '2026-02-17T15:18:54.280577+0000', 'last_unstale': '2026-02-17T15:19:40.974826+0000', 'last_undegraded': '2026-02-17T15:19:40.974826+0000', 'last_fullsized': '2026-02-17T15:19:40.974826+0000', 'mapping_epoch': 58, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 34299, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 109179, 'num_bytes_recovered': 894386748, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:19:42.605 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:19:47.606 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:19:47.607 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:19:47.607 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:19:47.608 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:19:47.693 INFO:teuthology.orchestra.run.trial024.stdout:98784247903 2026-02-17T15:19:47.694 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:47.695 INFO:teuthology.orchestra.run.trial024.stdout:171798691878 2026-02-17T15:19:47.695 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:47.698 INFO:teuthology.orchestra.run.trial024.stdout:214748364825 2026-02-17T15:19:47.698 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:47.700 INFO:teuthology.orchestra.run.trial024.stdout:236223201301 2026-02-17T15:19:47.701 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:47.844 INFO:teuthology.orchestra.run.trial024.stdout:171798691877 2026-02-17T15:19:47.848 INFO:teuthology.orchestra.run.trial024.stdout:98784247902 2026-02-17T15:19:47.857 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691878 got 171798691877 for osd.0 2026-02-17T15:19:47.861 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247903 got 98784247902 for osd.1 2026-02-17T15:19:47.889 INFO:teuthology.orchestra.run.trial024.stdout:214748364824 2026-02-17T15:19:47.900 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364825 got 214748364824 for osd.3 2026-02-17T15:19:47.904 INFO:teuthology.orchestra.run.trial024.stdout:236223201300 2026-02-17T15:19:47.916 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201301 got 236223201300 for osd.2 2026-02-17T15:19:48.858 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:48.861 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:48.901 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:48.917 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:49.027 INFO:teuthology.orchestra.run.trial024.stdout:171798691877 2026-02-17T15:19:49.040 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691878 got 171798691877 for osd.0 2026-02-17T15:19:49.058 INFO:teuthology.orchestra.run.trial024.stdout:214748364824 2026-02-17T15:19:49.064 INFO:teuthology.orchestra.run.trial024.stdout:98784247902 2026-02-17T15:19:49.071 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364825 got 214748364824 for osd.3 2026-02-17T15:19:49.077 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247903 got 98784247902 for osd.1 2026-02-17T15:19:49.111 INFO:teuthology.orchestra.run.trial024.stdout:236223201300 2026-02-17T15:19:49.125 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201301 got 236223201300 for osd.2 2026-02-17T15:19:50.041 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:50.072 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:50.079 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:50.126 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:50.194 INFO:teuthology.orchestra.run.trial024.stdout:171798691878 2026-02-17T15:19:50.207 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691878 got 171798691878 for osd.0 2026-02-17T15:19:50.207 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:50.240 INFO:teuthology.orchestra.run.trial024.stdout:214748364825 2026-02-17T15:19:50.244 INFO:teuthology.orchestra.run.trial024.stdout:98784247903 2026-02-17T15:19:50.252 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364825 got 214748364825 for osd.3 2026-02-17T15:19:50.253 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:50.257 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247903 got 98784247903 for osd.1 2026-02-17T15:19:50.257 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:50.286 INFO:teuthology.orchestra.run.trial024.stdout:236223201301 2026-02-17T15:19:50.298 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201301 got 236223201301 for osd.2 2026-02-17T15:19:50.299 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:50.299 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:19:50.485 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:19:50.486 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:19:50.496 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2026-02-17T15:19:49.507386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":28815,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114665,"num_bytes_recovered":939639356,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1199576,"kb_used_data":743508,"kb_used_omap":14,"kb_used_meta":455985,"kb_avail":281915944,"statfs":{"total":289910292480,"available":288681926656,"internally_reserved":0,"allocated":761352192,"data_stored":761091474,"data_compressed":13762,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14891,"internal_metadata":466929109},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8504,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8504,"num_bytes_recovered":69664768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001479"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":301043,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:19:47.684537+0000","last_change":"2026-02-17T15:18:54.281346+0000","last_active":"2026-02-17T15:19:47.684537+0000","last_peered":"2026-02-17T15:19:47.684537+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:19:47.684537+0000","last_undegraded":"2026-02-17T15:19:47.684537+0000","last_fullsized":"2026-02-17T15:19:47.684537+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":28815,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114663,"num_bytes_recovered":939311676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":48,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989512+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989512+0000","last_peered":"2026-02-17T15:18:53.989512+0000","last_clean":"2026-02-17T15:18:53.989512+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989512+0000","last_undegraded":"2026-02-17T15:18:53.989512+0000","last_fullsized":"2026-02-17T15:18:53.989512+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":28815,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114663,"num_bytes_recovered":939311676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1053892608,"data_stored":1053921280,"data_compressed":19480,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247903,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":293490688,"data_stored":293508742,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364825,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326560,"kb_used_data":286596,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045280,"statfs":{"total":96636764160,"available":96302366720,"internally_reserved":0,"allocated":293474304,"data_stored":293185158,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201301,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691878,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":401996,"kb_used_data":170300,"kb_used_omap":4,"kb_used_meta":231675,"kb_avail":93969844,"statfs":{"total":96636764160,"available":96225120256,"internally_reserved":0,"allocated":174387200,"data_stored":174397574,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":237235916},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":174948352,"data_stored":174952448,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:19:50.497 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 301043, 'reported_epoch': 59, 'state': 'active+remapped+backfilling', 'last_fresh': '2026-02-17T15:19:47.684537+0000', 'last_change': '2026-02-17T15:18:54.281346+0000', 'last_active': '2026-02-17T15:19:47.684537+0000', 'last_peered': '2026-02-17T15:19:47.684537+0000', 'last_clean': '2026-02-17T15:18:48.890765+0000', 'last_became_active': '2026-02-17T15:18:54.280577+0000', 'last_became_peered': '2026-02-17T15:18:54.280577+0000', 'last_unstale': '2026-02-17T15:19:47.684537+0000', 'last_undegraded': '2026-02-17T15:19:47.684537+0000', 'last_fullsized': '2026-02-17T15:19:47.684537+0000', 'mapping_epoch': 58, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 28815, 'num_objects_unfound': 0, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 114663, 'num_bytes_recovered': 939311676, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['3(0)', '1(1)', '2(2)'], 'object_location_counts': [{'shards': '1(1),2(2),3(0)', 'objects': 71529}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:19:50.497 DEBUG:tasks.ec_inconsistent_hinfo:backfilling 2026-02-17T15:19:54.698 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:19:54.690+0000 7fd32b3f6640 -1 osd.3 pg_epoch: 59 pg[2.0s0( v 26'71529 (26'71456,26'71529] local-lis/les=58/59 n=71529 ec=16/16 lis/c=58/55 les/c/f=59/56/0 sis=58) [3,1,0]/[3,1,2]p3(0) backfill=[0(2)] r=0 lpr=58 pi=[55,58)/1 crt=26'71529 lcod 0'0 mlcod 0'0 active+remapped+backfilling rops=1 mbc={0={},1={},2={}}] continue_recovery_op: 2:ad551702:::test:head has inconsistent hinfo 2026-02-17T15:19:54.698 INFO:tasks.ceph.osd.3.trial024.stderr:2026-02-17T15:19:54.690+0000 7fd32b3f6640 -1 log_channel(cluster) log [ERR] : 2.0s0 missing primary copy of 2:ad551702:::test:head, will try copies on 1(1),2(2) 2026-02-17T15:19:54.698 INFO:tasks.ceph.osd.2.trial024.stderr:2026-02-17T15:19:54.690+0000 7ff21c95e640 -1 log_channel(cluster) log [ERR] : Corruption detected: object 2:ad551702:::test:head is missing hash_info 2026-02-17T15:19:55.499 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:19:55.499 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:19:55.500 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:19:55.501 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:19:55.585 INFO:teuthology.orchestra.run.trial024.stdout:236223201303 2026-02-17T15:19:55.585 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:55.587 INFO:teuthology.orchestra.run.trial024.stdout:171798691880 2026-02-17T15:19:55.587 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:55.594 INFO:teuthology.orchestra.run.trial024.stdout:98784247906 2026-02-17T15:19:55.595 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:55.597 INFO:teuthology.orchestra.run.trial024.stdout:214748364827 2026-02-17T15:19:55.598 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:55.741 INFO:teuthology.orchestra.run.trial024.stdout:236223201302 2026-02-17T15:19:55.745 INFO:teuthology.orchestra.run.trial024.stdout:171798691879 2026-02-17T15:19:55.753 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201303 got 236223201302 for osd.2 2026-02-17T15:19:55.758 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691880 got 171798691879 for osd.0 2026-02-17T15:19:55.787 INFO:teuthology.orchestra.run.trial024.stdout:98784247904 2026-02-17T15:19:55.791 INFO:teuthology.orchestra.run.trial024.stdout:214748364826 2026-02-17T15:19:55.799 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247906 got 98784247904 for osd.1 2026-02-17T15:19:55.803 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364827 got 214748364826 for osd.3 2026-02-17T15:19:56.754 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:56.759 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:56.799 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:56.804 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:56.911 INFO:teuthology.orchestra.run.trial024.stdout:236223201302 2026-02-17T15:19:56.923 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201303 got 236223201302 for osd.2 2026-02-17T15:19:56.940 INFO:teuthology.orchestra.run.trial024.stdout:171798691879 2026-02-17T15:19:56.952 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691880 got 171798691879 for osd.0 2026-02-17T15:19:56.987 INFO:teuthology.orchestra.run.trial024.stdout:98784247905 2026-02-17T15:19:56.990 INFO:teuthology.orchestra.run.trial024.stdout:214748364826 2026-02-17T15:19:56.999 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247906 got 98784247905 for osd.1 2026-02-17T15:19:57.002 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364827 got 214748364826 for osd.3 2026-02-17T15:19:57.924 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:57.953 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:58.000 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:58.002 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:58.057 INFO:tasks.ceph.mon.a.trial024.stderr:2026-02-17T15:19:58.050+0000 7fc1d094f640 -1 log_channel(cluster) log [ERR] : Health check failed: Possible data damage: 1 pg backfill_unfound (PG_DAMAGED) 2026-02-17T15:19:58.083 INFO:teuthology.orchestra.run.trial024.stdout:236223201304 2026-02-17T15:19:58.095 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201303 got 236223201304 for osd.2 2026-02-17T15:19:58.096 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:58.111 INFO:teuthology.orchestra.run.trial024.stdout:171798691881 2026-02-17T15:19:58.123 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691880 got 171798691881 for osd.0 2026-02-17T15:19:58.123 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:58.161 INFO:teuthology.orchestra.run.trial024.stdout:98784247906 2026-02-17T15:19:58.163 INFO:teuthology.orchestra.run.trial024.stdout:214748364828 2026-02-17T15:19:58.174 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247906 got 98784247906 for osd.1 2026-02-17T15:19:58.174 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:58.175 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364827 got 214748364828 for osd.3 2026-02-17T15:19:58.175 DEBUG:teuthology.parallel:result is None 2026-02-17T15:19:58.176 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:19:58.364 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:19:58.365 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:19:58.374 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2026-02-17T15:19:57.508335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":23186,"num_objects_unfound":1,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120291,"num_bytes_recovered":985727548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1255068,"kb_used_data":767076,"kb_used_omap":14,"kb_used_meta":487921,"kb_avail":281860452,"statfs":{"total":289910292480,"available":288625102848,"internally_reserved":0,"allocated":785485824,"data_stored":785225106,"data_compressed":13762,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14891,"internal_metadata":499631573},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":-10052,"num_objects_unfound":1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10049,"num_bytes_recovered":82321408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001434"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":312296,"reported_epoch":59,"state":"active+backfill_unfound+remapped","last_fresh":"2026-02-17T15:19:54.695869+0000","last_change":"2026-02-17T15:19:54.697253+0000","last_active":"2026-02-17T15:19:54.695869+0000","last_peered":"2026-02-17T15:19:54.695869+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:19:54.695869+0000","last_undegraded":"2026-02-17T15:19:54.695869+0000","last_fullsized":"2026-02-17T15:19:54.695869+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":23186,"num_objects_unfound":1,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120289,"num_bytes_recovered":985399868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["1(1)"],"object_location_counts":[{"shards":"1(1)","objects":1},{"shards":"1(1),2(2),3(0)","objects":71528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":23186,"num_objects_unfound":1,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120289,"num_bytes_recovered":985399868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1076948992,"data_stored":1076977664,"data_compressed":19480,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247906,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":293490688,"data_stored":293508742,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364828,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326548,"kb_used_data":286596,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045292,"statfs":{"total":96636764160,"available":96302379008,"internally_reserved":0,"allocated":293474304,"data_stored":293185158,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201304,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691881,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457500,"kb_used_data":193868,"kb_used_omap":4,"kb_used_meta":263611,"kb_avail":93914340,"statfs":{"total":96636764160,"available":96168284160,"internally_reserved":0,"allocated":198520832,"data_stored":198531206,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":269938380},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":198004736,"data_stored":198008832,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:19:58.376 INFO:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 312296, 'reported_epoch': 59, 'state': 'active+backfill_unfound+remapped', 'last_fresh': '2026-02-17T15:19:54.695869+0000', 'last_change': '2026-02-17T15:19:54.697253+0000', 'last_active': '2026-02-17T15:19:54.695869+0000', 'last_peered': '2026-02-17T15:19:54.695869+0000', 'last_clean': '2026-02-17T15:18:48.890765+0000', 'last_became_active': '2026-02-17T15:18:54.280577+0000', 'last_became_peered': '2026-02-17T15:18:54.280577+0000', 'last_unstale': '2026-02-17T15:19:54.695869+0000', 'last_undegraded': '2026-02-17T15:19:54.695869+0000', 'last_fullsized': '2026-02-17T15:19:54.695869+0000', 'mapping_epoch': 58, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 23186, 'num_objects_unfound': 1, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 120289, 'num_bytes_recovered': 985399868, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['1(1)'], 'object_location_counts': [{'shards': '1(1)', 'objects': 1}, {'shards': '1(1),2(2),3(0)', 'objects': 71528}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:19:58.376 DEBUG:tasks.ec_inconsistent_hinfo:backfilled! 2026-02-17T15:19:58.376 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:19:58.377 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:19:58.377 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:19:58.378 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:19:58.461 INFO:teuthology.orchestra.run.trial024.stdout:236223201305 2026-02-17T15:19:58.461 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:58.463 INFO:teuthology.orchestra.run.trial024.stdout:214748364829 2026-02-17T15:19:58.464 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:58.468 INFO:teuthology.orchestra.run.trial024.stdout:171798691882 2026-02-17T15:19:58.469 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:58.473 INFO:teuthology.orchestra.run.trial024.stdout:98784247907 2026-02-17T15:19:58.474 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:58.614 INFO:teuthology.orchestra.run.trial024.stdout:236223201304 2026-02-17T15:19:58.617 INFO:teuthology.orchestra.run.trial024.stdout:214748364828 2026-02-17T15:19:58.627 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201305 got 236223201304 for osd.2 2026-02-17T15:19:58.629 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364829 got 214748364828 for osd.3 2026-02-17T15:19:58.657 INFO:teuthology.orchestra.run.trial024.stdout:171798691881 2026-02-17T15:19:58.662 INFO:teuthology.orchestra.run.trial024.stdout:98784247906 2026-02-17T15:19:58.668 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691882 got 171798691881 for osd.0 2026-02-17T15:19:58.674 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247907 got 98784247906 for osd.1 2026-02-17T15:19:59.628 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:19:59.630 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:19:59.669 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:19:59.674 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:19:59.780 INFO:teuthology.orchestra.run.trial024.stdout:214748364828 2026-02-17T15:19:59.789 INFO:teuthology.orchestra.run.trial024.stdout:236223201304 2026-02-17T15:19:59.792 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364829 got 214748364828 for osd.3 2026-02-17T15:19:59.801 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201305 got 236223201304 for osd.2 2026-02-17T15:19:59.825 INFO:teuthology.orchestra.run.trial024.stdout:171798691881 2026-02-17T15:19:59.835 INFO:teuthology.orchestra.run.trial024.stdout:98784247906 2026-02-17T15:19:59.837 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691882 got 171798691881 for osd.0 2026-02-17T15:19:59.847 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247907 got 98784247906 for osd.1 2026-02-17T15:20:00.001 INFO:tasks.ceph.mon.a.trial024.stderr:2026-02-17T15:19:59.994+0000 7fc1d094f640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1/71531 objects unfound (0.001%); Possible data damage: 1 pg backfill_unfound 2026-02-17T15:20:00.794 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:20:00.801 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:20:00.837 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:20:00.848 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:20:00.953 INFO:teuthology.orchestra.run.trial024.stdout:214748364829 2026-02-17T15:20:00.965 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364829 got 214748364829 for osd.3 2026-02-17T15:20:00.965 DEBUG:teuthology.parallel:result is None 2026-02-17T15:20:00.982 INFO:teuthology.orchestra.run.trial024.stdout:171798691882 2026-02-17T15:20:00.982 INFO:teuthology.orchestra.run.trial024.stdout:236223201305 2026-02-17T15:20:00.994 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691882 got 171798691882 for osd.0 2026-02-17T15:20:00.995 DEBUG:teuthology.parallel:result is None 2026-02-17T15:20:00.995 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201305 got 236223201305 for osd.2 2026-02-17T15:20:00.996 DEBUG:teuthology.parallel:result is None 2026-02-17T15:20:01.029 INFO:teuthology.orchestra.run.trial024.stdout:98784247907 2026-02-17T15:20:01.040 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247907 got 98784247907 for osd.1 2026-02-17T15:20:01.041 DEBUG:teuthology.parallel:result is None 2026-02-17T15:20:01.041 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:01.233 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:01.234 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:01.244 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2026-02-17T15:19:59.508619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":23186,"num_objects_unfound":1,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120291,"num_bytes_recovered":985727548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1255068,"kb_used_data":767076,"kb_used_omap":14,"kb_used_meta":487921,"kb_avail":281860452,"statfs":{"total":289910292480,"available":288625102848,"internally_reserved":0,"allocated":785485824,"data_stored":785225106,"data_compressed":13762,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14891,"internal_metadata":499631573},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":-5978,"num_objects_unfound":1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5975,"num_bytes_recovered":48947200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001484"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":312296,"reported_epoch":59,"state":"active+backfill_unfound+remapped","last_fresh":"2026-02-17T15:19:54.695869+0000","last_change":"2026-02-17T15:19:54.697253+0000","last_active":"2026-02-17T15:19:54.695869+0000","last_peered":"2026-02-17T15:19:54.695869+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:19:54.695869+0000","last_undegraded":"2026-02-17T15:19:54.695869+0000","last_fullsized":"2026-02-17T15:19:54.695869+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":23186,"num_objects_unfound":1,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120289,"num_bytes_recovered":985399868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["1(1)"],"object_location_counts":[{"shards":"1(1)","objects":1},{"shards":"1(1),2(2),3(0)","objects":71528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":23186,"num_objects_unfound":1,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120289,"num_bytes_recovered":985399868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1076948992,"data_stored":1076977664,"data_compressed":19480,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247907,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":293490688,"data_stored":293508742,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326548,"kb_used_data":286596,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045292,"statfs":{"total":96636764160,"available":96302379008,"internally_reserved":0,"allocated":293474304,"data_stored":293185158,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201305,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691882,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457500,"kb_used_data":193868,"kb_used_omap":4,"kb_used_meta":263611,"kb_avail":93914340,"statfs":{"total":96636764160,"available":96168284160,"internally_reserved":0,"allocated":198520832,"data_stored":198531206,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":269938380},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":198004736,"data_stored":198008832,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:01.246 DEBUG:tasks.ec_inconsistent_hinfo:pg={'pgid': '2.0', 'version': "26'71529", 'reported_seq': 312296, 'reported_epoch': 59, 'state': 'active+backfill_unfound+remapped', 'last_fresh': '2026-02-17T15:19:54.695869+0000', 'last_change': '2026-02-17T15:19:54.697253+0000', 'last_active': '2026-02-17T15:19:54.695869+0000', 'last_peered': '2026-02-17T15:19:54.695869+0000', 'last_clean': '2026-02-17T15:18:48.890765+0000', 'last_became_active': '2026-02-17T15:18:54.280577+0000', 'last_became_peered': '2026-02-17T15:18:54.280577+0000', 'last_unstale': '2026-02-17T15:19:54.695869+0000', 'last_undegraded': '2026-02-17T15:19:54.695869+0000', 'last_fullsized': '2026-02-17T15:19:54.695869+0000', 'mapping_epoch': 58, 'log_start': "26'71456", 'ondisk_log_start': "26'71456", 'created': 16, 'last_epoch_clean': 56, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "19'2", 'last_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_deep_scrub': "19'2", 'last_deep_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'last_clean_scrub_stamp': '2026-02-17T15:14:39.584543+0000', 'objects_scrubbed': 2, 'log_size': 73, 'log_dups_size': 3000, 'ondisk_log_size': 73, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 585957674, 'num_objects': 71529, 'num_object_clones': 0, 'num_object_copies': 214587, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 23186, 'num_objects_unfound': 1, 'num_objects_dirty': 71529, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 71529, 'num_write_kb': 572227, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 120289, 'num_bytes_recovered': 985399868, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 1}, 'up': [3, 1, 0], 'acting': [3, 1, 2], 'avail_no_missing': ['1(1)'], 'object_location_counts': [{'shards': '1(1)', 'objects': 1}, {'shards': '1(1),2(2),3(0)', 'objects': 71528}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2026-02-17T15:20:01.246 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2026-02-17T15:20:01.428 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:01.428 INFO:teuthology.orchestra.run.trial024.stdout:{"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","health":{"status":"HEALTH_ERR","checks":{"OBJECT_UNFOUND":{"severity":"HEALTH_WARN","summary":{"message":"1/71531 objects unfound (0.001%)","count":1},"muted":false},"PG_DAMAGED":{"severity":"HEALTH_ERR","summary":{"message":"Possible data damage: 1 pg backfill_unfound","count":1},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":374,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":59,"num_osds":4,"num_up_osds":4,"osd_up_since":1771341527,"num_in_osds":3,"osd_in_since":1771341531,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":1},{"state_name":"active+backfill_unfound+remapped","count":1}],"num_pgs":2,"num_pools":2,"num_objects":71531,"data_bytes":586285866,"bytes_used":1285189632,"bytes_avail":288625102848,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":214591,"degraded_ratio":9.3200553611288457e-06,"misplaced_objects":23186,"misplaced_total":214591,"misplaced_ratio":0.1080474018015667,"unfound_objects":1,"unfound_total":71531,"unfound_ratio":1.3979952747759712e-05,"recovering_objects_per_sec":497,"recovering_bytes_per_sec":4078428,"recovering_keys_per_sec":0,"num_objects_recovered":5975,"num_bytes_recovered":48947200,"num_keys_recovered":0},"fsmap":{"epoch":1,"btime":"2026-02-17T15:13:47:232255+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2026-02-17T15:19:47.507175+0000","services":{}},"progress_events":{}} 2026-02-17T15:20:01.439 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:{'fsid': 'ac9bccef-327a-4812-a379-08d174e6a4bc', 'health': {'status': 'HEALTH_ERR', 'checks': {'OBJECT_UNFOUND': {'severity': 'HEALTH_WARN', 'summary': {'message': '1/71531 objects unfound (0.001%)', 'count': 1}, 'muted': False}, 'PG_DAMAGED': {'severity': 'HEALTH_ERR', 'summary': {'message': 'Possible data damage: 1 pg backfill_unfound', 'count': 1}, 'muted': False}}, 'mutes': []}, 'election_epoch': 10, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 374, 'monmap': {'epoch': 1, 'min_mon_release_name': 'squid', 'num_mons': 3}, 'osdmap': {'epoch': 59, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1771341527, 'num_in_osds': 3, 'osd_in_since': 1771341531, 'num_remapped_pgs': 1}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+clean', 'count': 1}, {'state_name': 'active+backfill_unfound+remapped', 'count': 1}], 'num_pgs': 2, 'num_pools': 2, 'num_objects': 71531, 'data_bytes': 586285866, 'bytes_used': 1285189632, 'bytes_avail': 288625102848, 'bytes_total': 289910292480, 'degraded_objects': 2, 'degraded_total': 214591, 'degraded_ratio': 9.320055361128846e-06, 'misplaced_objects': 23186, 'misplaced_total': 214591, 'misplaced_ratio': 0.1080474018015667, 'unfound_objects': 1, 'unfound_total': 71531, 'unfound_ratio': 1.3979952747759712e-05, 'recovering_objects_per_sec': 497, 'recovering_bytes_per_sec': 4078428, 'recovering_keys_per_sec': 0, 'num_objects_recovered': 5975, 'num_bytes_recovered': 48947200, 'num_keys_recovered': 0}, 'fsmap': {'epoch': 1, 'btime': '2026-02-17T15:13:47:232255+0000', 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 8, 'modified': '2026-02-17T15:19:47.507175+0000', 'services': {}}, 'progress_events': {}} 2026-02-17T15:20:01.439 DEBUG:tasks.ec_inconsistent_hinfo:there are 1 unfound objects 2026-02-17T15:20:01.440 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- pg 2.0 list_unfound '{}' 2026-02-17T15:20:01.514 INFO:teuthology.orchestra.run.trial024.stdout:{ 2026-02-17T15:20:01.514 INFO:teuthology.orchestra.run.trial024.stdout: "offset": { 2026-02-17T15:20:01.514 INFO:teuthology.orchestra.run.trial024.stdout: "oid": "", 2026-02-17T15:20:01.514 INFO:teuthology.orchestra.run.trial024.stdout: "key": "", 2026-02-17T15:20:01.514 INFO:teuthology.orchestra.run.trial024.stdout: "snapid": 0, 2026-02-17T15:20:01.514 INFO:teuthology.orchestra.run.trial024.stdout: "hash": 0, 2026-02-17T15:20:01.514 INFO:teuthology.orchestra.run.trial024.stdout: "max": 0, 2026-02-17T15:20:01.514 INFO:teuthology.orchestra.run.trial024.stdout: "pool": -9223372036854775808, 2026-02-17T15:20:01.515 INFO:teuthology.orchestra.run.trial024.stdout: "namespace": "" 2026-02-17T15:20:01.515 INFO:teuthology.orchestra.run.trial024.stdout: }, 2026-02-17T15:20:01.515 INFO:teuthology.orchestra.run.trial024.stdout: "num_missing": 1, 2026-02-17T15:20:01.515 INFO:teuthology.orchestra.run.trial024.stdout: "num_unfound": 1, 2026-02-17T15:20:01.515 INFO:teuthology.orchestra.run.trial024.stdout: "objects": [ 2026-02-17T15:20:01.515 INFO:teuthology.orchestra.run.trial024.stdout: { 2026-02-17T15:20:01.515 INFO:teuthology.orchestra.run.trial024.stdout: "oid": { 2026-02-17T15:20:01.515 INFO:teuthology.orchestra.run.trial024.stdout: "oid": "test", 2026-02-17T15:20:01.515 INFO:teuthology.orchestra.run.trial024.stdout: "key": "", 2026-02-17T15:20:01.515 INFO:teuthology.orchestra.run.trial024.stdout: "snapid": -2, 2026-02-17T15:20:01.515 INFO:teuthology.orchestra.run.trial024.stdout: "hash": 1088989877, 2026-02-17T15:20:01.515 INFO:teuthology.orchestra.run.trial024.stdout: "max": 0, 2026-02-17T15:20:01.516 INFO:teuthology.orchestra.run.trial024.stdout: "pool": 2, 2026-02-17T15:20:01.516 INFO:teuthology.orchestra.run.trial024.stdout: "namespace": "" 2026-02-17T15:20:01.516 INFO:teuthology.orchestra.run.trial024.stdout: }, 2026-02-17T15:20:01.516 INFO:teuthology.orchestra.run.trial024.stdout: "need": "19'2", 2026-02-17T15:20:01.516 INFO:teuthology.orchestra.run.trial024.stdout: "have": "0'0", 2026-02-17T15:20:01.516 INFO:teuthology.orchestra.run.trial024.stdout: "flags": "none", 2026-02-17T15:20:01.516 INFO:teuthology.orchestra.run.trial024.stdout: "clean_regions": "clean_offsets: [], clean_omap: false, new_object: true", 2026-02-17T15:20:01.516 INFO:teuthology.orchestra.run.trial024.stdout: "locations": [ 2026-02-17T15:20:01.516 INFO:teuthology.orchestra.run.trial024.stdout: "1(1)" 2026-02-17T15:20:01.516 INFO:teuthology.orchestra.run.trial024.stdout: ] 2026-02-17T15:20:01.516 INFO:teuthology.orchestra.run.trial024.stdout: } 2026-02-17T15:20:01.516 INFO:teuthology.orchestra.run.trial024.stdout: ], 2026-02-17T15:20:01.517 INFO:teuthology.orchestra.run.trial024.stdout: "state": "NotBackfilling", 2026-02-17T15:20:01.517 INFO:teuthology.orchestra.run.trial024.stdout: "available_might_have_unfound": true, 2026-02-17T15:20:01.517 INFO:teuthology.orchestra.run.trial024.stdout: "might_have_unfound": [], 2026-02-17T15:20:01.517 INFO:teuthology.orchestra.run.trial024.stdout: "more": false 2026-02-17T15:20:01.517 INFO:teuthology.orchestra.run.trial024.stdout:} 2026-02-17T15:20:01.523 DEBUG:tasks.ec_inconsistent_hinfo:list_pg_unfound={'offset': {'oid': '', 'key': '', 'snapid': 0, 'hash': 0, 'max': 0, 'pool': -9223372036854775808, 'namespace': ''}, 'num_missing': 1, 'num_unfound': 1, 'objects': [{'oid': {'oid': 'test', 'key': '', 'snapid': -2, 'hash': 1088989877, 'max': 0, 'pool': 2, 'namespace': ''}, 'need': "19'2", 'have': "0'0", 'flags': 'none', 'clean_regions': 'clean_offsets: [], clean_omap: false, new_object: true', 'locations': ['1(1)']}], 'state': 'NotBackfilling', 'available_might_have_unfound': True, 'might_have_unfound': []} 2026-02-17T15:20:01.523 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:01.676 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:01.676 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:01.686 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":239,"stamp":"2026-02-17T15:20:01.508972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":23186,"num_objects_unfound":1,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120291,"num_bytes_recovered":985727548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1255068,"kb_used_data":767076,"kb_used_omap":14,"kb_used_meta":487921,"kb_avail":281860452,"statfs":{"total":289910292480,"available":288625102848,"internally_reserved":0,"allocated":785485824,"data_stored":785225106,"data_compressed":13762,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14891,"internal_metadata":499631573},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":-5629,"num_objects_unfound":1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5626,"num_bytes_recovered":46088192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001585"},"pg_stats":[{"pgid":"2.0","version":"26'71529","reported_seq":312296,"reported_epoch":59,"state":"active+backfill_unfound+remapped","last_fresh":"2026-02-17T15:19:54.695869+0000","last_change":"2026-02-17T15:19:54.697253+0000","last_active":"2026-02-17T15:19:54.695869+0000","last_peered":"2026-02-17T15:19:54.695869+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:19:54.695869+0000","last_undegraded":"2026-02-17T15:19:54.695869+0000","last_fullsized":"2026-02-17T15:19:54.695869+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":73,"log_dups_size":3000,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":23186,"num_objects_unfound":1,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120289,"num_bytes_recovered":985399868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["1(1)"],"object_location_counts":[{"shards":"1(1)","objects":1},{"shards":"1(1),2(2),3(0)","objects":71528}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":1,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":23186,"num_objects_unfound":1,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120289,"num_bytes_recovered":985399868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1076948992,"data_stored":1076977664,"data_compressed":19480,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1696,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247908,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":293490688,"data_stored":293508742,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364829,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326548,"kb_used_data":286596,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045292,"statfs":{"total":96636764160,"available":96302379008,"internally_reserved":0,"allocated":293474304,"data_stored":293185158,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201305,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691883,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457500,"kb_used_data":193868,"kb_used_omap":4,"kb_used_meta":263611,"kb_avail":93914340,"statfs":{"total":96636764160,"available":96168284160,"internally_reserved":0,"allocated":198520832,"data_stored":198531206,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":269938380},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":198004736,"data_stored":198008832,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292986880,"data_compressed":4260,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":6904,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292982784,"data_stored":292990976,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:01.687 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg 2.0 mark_unfound_lost delete 2026-02-17T15:20:01.769 INFO:teuthology.orchestra.run.trial024.stderr:pg has 1 objects unfound and apparently lost marking 2026-02-17T15:20:01.781 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2026-02-17T15:20:01.782 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2026-02-17T15:20:01.782 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2026-02-17T15:20:01.783 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2026-02-17T15:20:01.870 INFO:teuthology.orchestra.run.trial024.stdout:214748364830 2026-02-17T15:20:01.871 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:20:01.873 INFO:teuthology.orchestra.run.trial024.stdout:171798691884 2026-02-17T15:20:01.873 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:20:01.875 INFO:teuthology.orchestra.run.trial024.stdout:236223201307 2026-02-17T15:20:01.876 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:20:01.880 INFO:teuthology.orchestra.run.trial024.stdout:98784247909 2026-02-17T15:20:01.881 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:20:02.030 INFO:teuthology.orchestra.run.trial024.stdout:171798691882 2026-02-17T15:20:02.038 INFO:teuthology.orchestra.run.trial024.stdout:214748364829 2026-02-17T15:20:02.044 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691884 got 171798691882 for osd.0 2026-02-17T15:20:02.051 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364830 got 214748364829 for osd.3 2026-02-17T15:20:02.070 INFO:teuthology.orchestra.run.trial024.stdout:98784247908 2026-02-17T15:20:02.076 INFO:teuthology.orchestra.run.trial024.stdout:236223201305 2026-02-17T15:20:02.083 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247909 got 98784247908 for osd.1 2026-02-17T15:20:02.088 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201307 got 236223201305 for osd.2 2026-02-17T15:20:03.045 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:20:03.051 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:20:03.083 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:20:03.089 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:20:03.192 INFO:teuthology.orchestra.run.trial024.stdout:171798691883 2026-02-17T15:20:03.204 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691884 got 171798691883 for osd.0 2026-02-17T15:20:03.241 INFO:teuthology.orchestra.run.trial024.stdout:236223201305 2026-02-17T15:20:03.241 INFO:teuthology.orchestra.run.trial024.stdout:214748364829 2026-02-17T15:20:03.253 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201307 got 236223201305 for osd.2 2026-02-17T15:20:03.254 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364830 got 214748364829 for osd.3 2026-02-17T15:20:03.256 INFO:teuthology.orchestra.run.trial024.stdout:98784247908 2026-02-17T15:20:03.269 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247909 got 98784247908 for osd.1 2026-02-17T15:20:04.206 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2026-02-17T15:20:04.254 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2026-02-17T15:20:04.255 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2026-02-17T15:20:04.270 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2026-02-17T15:20:04.352 INFO:teuthology.orchestra.run.trial024.stdout:171798691884 2026-02-17T15:20:04.364 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 171798691884 got 171798691884 for osd.0 2026-02-17T15:20:04.364 DEBUG:teuthology.parallel:result is None 2026-02-17T15:20:04.417 INFO:teuthology.orchestra.run.trial024.stdout:214748364831 2026-02-17T15:20:04.419 INFO:teuthology.orchestra.run.trial024.stdout:236223201307 2026-02-17T15:20:04.430 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 214748364830 got 214748364831 for osd.3 2026-02-17T15:20:04.430 DEBUG:teuthology.parallel:result is None 2026-02-17T15:20:04.433 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 236223201307 got 236223201307 for osd.2 2026-02-17T15:20:04.434 DEBUG:teuthology.parallel:result is None 2026-02-17T15:20:04.451 INFO:teuthology.orchestra.run.trial024.stdout:98784247909 2026-02-17T15:20:04.462 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:need seq 98784247909 got 98784247909 for osd.1 2026-02-17T15:20:04.462 DEBUG:teuthology.parallel:result is None 2026-02-17T15:20:04.462 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:waiting for recovery to complete 2026-02-17T15:20:04.462 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:04.651 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:04.652 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:04.662 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2026-02-17T15:20:03.509236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22793,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120687,"num_bytes_recovered":988963388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1255068,"kb_used_data":767076,"kb_used_omap":14,"kb_used_meta":487921,"kb_avail":281860452,"statfs":{"total":289910292480,"available":288625102848,"internally_reserved":0,"allocated":785485824,"data_stored":785225106,"data_compressed":13762,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14891,"internal_metadata":499631573},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6022,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6022,"num_bytes_recovered":49324032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001618"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":313095,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:02.236535+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:02.236535+0000","last_peered":"2026-02-17T15:20:02.236535+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:02.236535+0000","last_undegraded":"2026-02-17T15:20:02.236535+0000","last_fullsized":"2026-02-17T15:20:02.236535+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22793,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120685,"num_bytes_recovered":988635708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22793,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120685,"num_bytes_recovered":988635708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1077239808,"data_stored":1077256192,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247909,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":293490688,"data_stored":293508742,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326548,"kb_used_data":286596,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045292,"statfs":{"total":96636764160,"available":96302379008,"internally_reserved":0,"allocated":293474304,"data_stored":293185158,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201307,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691884,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457500,"kb_used_data":193868,"kb_used_omap":4,"kb_used_meta":263611,"kb_avail":93914340,"statfs":{"total":96636764160,"available":96168284160,"internally_reserved":0,"allocated":198520832,"data_stored":198531206,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":269938380},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":198307840,"data_stored":198311936,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:04.663 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:04.805 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:04.805 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:04.815 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2026-02-17T15:20:03.509236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22793,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120687,"num_bytes_recovered":988963388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1255068,"kb_used_data":767076,"kb_used_omap":14,"kb_used_meta":487921,"kb_avail":281860452,"statfs":{"total":289910292480,"available":288625102848,"internally_reserved":0,"allocated":785485824,"data_stored":785225106,"data_compressed":13762,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14891,"internal_metadata":499631573},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6022,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6022,"num_bytes_recovered":49324032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001618"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":313095,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:02.236535+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:02.236535+0000","last_peered":"2026-02-17T15:20:02.236535+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:02.236535+0000","last_undegraded":"2026-02-17T15:20:02.236535+0000","last_fullsized":"2026-02-17T15:20:02.236535+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22793,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120685,"num_bytes_recovered":988635708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22793,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120685,"num_bytes_recovered":988635708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1077239808,"data_stored":1077256192,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247909,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":293490688,"data_stored":293508742,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326548,"kb_used_data":286596,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045292,"statfs":{"total":96636764160,"available":96302379008,"internally_reserved":0,"allocated":293474304,"data_stored":293185158,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201307,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691884,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457500,"kb_used_data":193868,"kb_used_omap":4,"kb_used_meta":263611,"kb_avail":93914340,"statfs":{"total":96636764160,"available":96168284160,"internally_reserved":0,"allocated":198520832,"data_stored":198531206,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":269938380},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":198307840,"data_stored":198311936,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:04.816 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:04.957 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:04.957 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:04.967 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2026-02-17T15:20:03.509236+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22793,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120687,"num_bytes_recovered":988963388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1255068,"kb_used_data":767076,"kb_used_omap":14,"kb_used_meta":487921,"kb_avail":281860452,"statfs":{"total":289910292480,"available":288625102848,"internally_reserved":0,"allocated":785485824,"data_stored":785225106,"data_compressed":13762,"data_compressed_allocated":61440,"data_compressed_original":696320,"omap_allocated":14891,"internal_metadata":499631573},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6022,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6022,"num_bytes_recovered":49324032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001618"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":313095,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:02.236535+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:02.236535+0000","last_peered":"2026-02-17T15:20:02.236535+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:02.236535+0000","last_undegraded":"2026-02-17T15:20:02.236535+0000","last_fullsized":"2026-02-17T15:20:02.236535+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22793,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120685,"num_bytes_recovered":988635708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":22793,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120685,"num_bytes_recovered":988635708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1077239808,"data_stored":1077256192,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247909,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471020,"kb_used_data":286612,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900820,"statfs":{"total":96636764160,"available":96154439680,"internally_reserved":0,"allocated":293490688,"data_stored":293508742,"data_compressed":4853,"data_compressed_allocated":28672,"data_compressed_original":344064,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364831,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326548,"kb_used_data":286596,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045292,"statfs":{"total":96636764160,"available":96302379008,"internally_reserved":0,"allocated":293474304,"data_stored":293185158,"data_compressed":4864,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201307,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691884,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":457500,"kb_used_data":193868,"kb_used_omap":4,"kb_used_meta":263611,"kb_avail":93914340,"statfs":{"total":96636764160,"available":96168284160,"internally_reserved":0,"allocated":198520832,"data_stored":198531206,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":269938380},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":198307840,"data_stored":198311936,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:07.968 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:08.115 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:08.116 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:08.125 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2026-02-17T15:20:07.509692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18727,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124753,"num_bytes_recovered":1022263892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294820,"kb_used_data":779060,"kb_used_omap":14,"kb_used_meta":515697,"kb_avail":281820700,"statfs":{"total":289910292480,"available":288584396800,"internally_reserved":0,"allocated":797757440,"data_stored":797488530,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":528074197},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6427,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6427,"num_bytes_recovered":52633624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001585"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":321227,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:07.237127+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:07.237127+0000","last_peered":"2026-02-17T15:20:07.237127+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:07.237127+0000","last_undegraded":"2026-02-17T15:20:07.237127+0000","last_fullsized":"2026-02-17T15:20:07.237127+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18727,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124751,"num_bytes_recovered":1021936212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18727,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124751,"num_bytes_recovered":1021936212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089900544,"data_stored":1089916928,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247910,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201308,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691885,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497268,"kb_used_data":205860,"kb_used_omap":4,"kb_used_meta":291387,"kb_avail":93874572,"statfs":{"total":96636764160,"available":96127561728,"internally_reserved":0,"allocated":210800640,"data_stored":210811014,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":298381004},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":210968576,"data_stored":210972672,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:08.127 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:08.272 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:08.272 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:08.282 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2026-02-17T15:20:07.509692+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18727,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124753,"num_bytes_recovered":1022263892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294820,"kb_used_data":779060,"kb_used_omap":14,"kb_used_meta":515697,"kb_avail":281820700,"statfs":{"total":289910292480,"available":288584396800,"internally_reserved":0,"allocated":797757440,"data_stored":797488530,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":528074197},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6427,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6427,"num_bytes_recovered":52633624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001585"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":321227,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:07.237127+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:07.237127+0000","last_peered":"2026-02-17T15:20:07.237127+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:07.237127+0000","last_undegraded":"2026-02-17T15:20:07.237127+0000","last_fullsized":"2026-02-17T15:20:07.237127+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18727,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124751,"num_bytes_recovered":1021936212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18727,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124751,"num_bytes_recovered":1021936212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089900544,"data_stored":1089916928,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247910,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201308,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691885,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497268,"kb_used_data":205860,"kb_used_omap":4,"kb_used_meta":291387,"kb_avail":93874572,"statfs":{"total":96636764160,"available":96127561728,"internally_reserved":0,"allocated":210800640,"data_stored":210811014,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":298381004},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":210968576,"data_stored":210972672,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:11.284 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:11.428 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:11.428 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:11.439 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2026-02-17T15:20:09.509928+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18727,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124753,"num_bytes_recovered":1022263892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1294820,"kb_used_data":779060,"kb_used_omap":14,"kb_used_meta":515697,"kb_avail":281820700,"statfs":{"total":289910292480,"available":288584396800,"internally_reserved":0,"allocated":797757440,"data_stored":797488530,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":528074197},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-1,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-4459,"num_objects_unfound":-1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4462,"num_bytes_recovered":36536344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001593"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":321227,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:07.237127+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:07.237127+0000","last_peered":"2026-02-17T15:20:07.237127+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:07.237127+0000","last_undegraded":"2026-02-17T15:20:07.237127+0000","last_fullsized":"2026-02-17T15:20:07.237127+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18727,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124751,"num_bytes_recovered":1021936212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18727,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124751,"num_bytes_recovered":1021936212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089900544,"data_stored":1089916928,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247910,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201308,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691885,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":497268,"kb_used_data":205860,"kb_used_omap":4,"kb_used_meta":291387,"kb_avail":93874572,"statfs":{"total":96636764160,"available":96127561728,"internally_reserved":0,"allocated":210800640,"data_stored":210811014,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":298381004},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":210968576,"data_stored":210972672,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:11.440 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:11.583 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:11.583 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:11.595 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2026-02-17T15:20:11.510153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18727,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124753,"num_bytes_recovered":1022263892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1310776,"kb_used_data":795016,"kb_used_omap":14,"kb_used_meta":515697,"kb_avail":281804744,"statfs":{"total":289910292480,"available":288568057856,"internally_reserved":0,"allocated":814096384,"data_stored":813827474,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":528074197},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-1,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-4459,"num_objects_unfound":-1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4462,"num_bytes_recovered":36536344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001533"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":321227,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:07.237127+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:07.237127+0000","last_peered":"2026-02-17T15:20:07.237127+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:07.237127+0000","last_undegraded":"2026-02-17T15:20:07.237127+0000","last_fullsized":"2026-02-17T15:20:07.237127+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18727,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124751,"num_bytes_recovered":1021936212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":18727,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":124751,"num_bytes_recovered":1021936212,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1106104320,"data_stored":1106120704,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364832,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201308,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691886,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513224,"kb_used_data":221816,"kb_used_omap":4,"kb_used_meta":291387,"kb_avail":93858616,"statfs":{"total":96636764160,"available":96111222784,"internally_reserved":0,"allocated":227139584,"data_stored":227149958,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":298381004},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":227172352,"data_stored":227176448,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:14.597 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:14.743 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:14.744 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:14.754 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2026-02-17T15:20:13.510359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14755,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128725,"num_bytes_recovered":1054802516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1310776,"kb_used_data":795016,"kb_used_omap":14,"kb_used_meta":515697,"kb_avail":281804744,"statfs":{"total":289910292480,"available":288568057856,"internally_reserved":0,"allocated":814096384,"data_stored":813827474,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":528074197},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-1,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-8431,"num_objects_unfound":-1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8434,"num_bytes_recovered":69074968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001386"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":329171,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:12.237164+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:12.237164+0000","last_peered":"2026-02-17T15:20:12.237164+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:12.237164+0000","last_undegraded":"2026-02-17T15:20:12.237164+0000","last_fullsized":"2026-02-17T15:20:12.237164+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14755,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128723,"num_bytes_recovered":1054474836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14755,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128723,"num_bytes_recovered":1054474836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1106104320,"data_stored":1106120704,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201309,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691886,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513224,"kb_used_data":221816,"kb_used_omap":4,"kb_used_meta":291387,"kb_avail":93858616,"statfs":{"total":96636764160,"available":96111222784,"internally_reserved":0,"allocated":227139584,"data_stored":227149958,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":298381004},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":227172352,"data_stored":227176448,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:14.755 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:14.907 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:14.907 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:14.918 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2026-02-17T15:20:13.510359+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14755,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128725,"num_bytes_recovered":1054802516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1310776,"kb_used_data":795016,"kb_used_omap":14,"kb_used_meta":515697,"kb_avail":281804744,"statfs":{"total":289910292480,"available":288568057856,"internally_reserved":0,"allocated":814096384,"data_stored":813827474,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":528074197},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-1,"num_objects_missing":0,"num_objects_degraded":-2,"num_objects_misplaced":-8431,"num_objects_unfound":-1,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8434,"num_bytes_recovered":69074968,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001386"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":329171,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:12.237164+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:12.237164+0000","last_peered":"2026-02-17T15:20:12.237164+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:12.237164+0000","last_undegraded":"2026-02-17T15:20:12.237164+0000","last_fullsized":"2026-02-17T15:20:12.237164+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14755,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128723,"num_bytes_recovered":1054474836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14755,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128723,"num_bytes_recovered":1054474836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1106104320,"data_stored":1106120704,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247911,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364833,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201309,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691886,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":513224,"kb_used_data":221816,"kb_used_omap":4,"kb_used_meta":291387,"kb_avail":93858616,"statfs":{"total":96636764160,"available":96111222784,"internally_reserved":0,"allocated":227139584,"data_stored":227149958,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":298381004},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":227172352,"data_stored":227176448,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:17.921 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:18.078 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:18.079 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:18.089 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2026-02-17T15:20:17.510846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10712,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132768,"num_bytes_recovered":1087922772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1362788,"kb_used_data":810932,"kb_used_omap":14,"kb_used_meta":551793,"kb_avail":281752732,"statfs":{"total":289910292480,"available":288514797568,"internally_reserved":0,"allocated":830394368,"data_stored":830125458,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":565036501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-12081,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12081,"num_bytes_recovered":98959384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001373"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":337257,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:17.238160+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:17.238160+0000","last_peered":"2026-02-17T15:20:17.238160+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:17.238160+0000","last_undegraded":"2026-02-17T15:20:17.238160+0000","last_fullsized":"2026-02-17T15:20:17.238160+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10712,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132766,"num_bytes_recovered":1087595092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10712,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132766,"num_bytes_recovered":1087595092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1122332672,"data_stored":1122349056,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201310,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691887,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565236,"kb_used_data":237732,"kb_used_omap":4,"kb_used_meta":327483,"kb_avail":93806604,"statfs":{"total":96636764160,"available":96057962496,"internally_reserved":0,"allocated":243437568,"data_stored":243447942,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":335343308},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":243400704,"data_stored":243404800,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:18.090 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:18.236 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:18.237 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:18.248 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2026-02-17T15:20:17.510846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10712,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132768,"num_bytes_recovered":1087922772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1362788,"kb_used_data":810932,"kb_used_omap":14,"kb_used_meta":551793,"kb_avail":281752732,"statfs":{"total":289910292480,"available":288514797568,"internally_reserved":0,"allocated":830394368,"data_stored":830125458,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":565036501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-12081,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12081,"num_bytes_recovered":98959384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001373"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":337257,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:17.238160+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:17.238160+0000","last_peered":"2026-02-17T15:20:17.238160+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:17.238160+0000","last_undegraded":"2026-02-17T15:20:17.238160+0000","last_fullsized":"2026-02-17T15:20:17.238160+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10712,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132766,"num_bytes_recovered":1087595092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10712,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132766,"num_bytes_recovered":1087595092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1122332672,"data_stored":1122349056,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201310,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691887,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565236,"kb_used_data":237732,"kb_used_omap":4,"kb_used_meta":327483,"kb_avail":93806604,"statfs":{"total":96636764160,"available":96057962496,"internally_reserved":0,"allocated":243437568,"data_stored":243447942,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":335343308},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":243400704,"data_stored":243404800,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:21.250 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:21.396 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:21.396 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:21.406 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2026-02-17T15:20:19.511078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10712,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132768,"num_bytes_recovered":1087922772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1362788,"kb_used_data":810932,"kb_used_omap":14,"kb_used_meta":551793,"kb_avail":281752732,"statfs":{"total":289910292480,"available":288514797568,"internally_reserved":0,"allocated":830394368,"data_stored":830125458,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":565036501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8015,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8015,"num_bytes_recovered":65658880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001386"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":337257,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:17.238160+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:17.238160+0000","last_peered":"2026-02-17T15:20:17.238160+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:17.238160+0000","last_undegraded":"2026-02-17T15:20:17.238160+0000","last_fullsized":"2026-02-17T15:20:17.238160+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10712,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132766,"num_bytes_recovered":1087595092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10712,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132766,"num_bytes_recovered":1087595092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1122332672,"data_stored":1122349056,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247912,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201310,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691887,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":565236,"kb_used_data":237732,"kb_used_omap":4,"kb_used_meta":327483,"kb_avail":93806604,"statfs":{"total":96636764160,"available":96057962496,"internally_reserved":0,"allocated":243437568,"data_stored":243447942,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":335343308},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":243400704,"data_stored":243404800,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:21.407 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:21.547 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:21.548 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:21.558 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2026-02-17T15:20:21.511310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10712,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132768,"num_bytes_recovered":1087922772,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1379340,"kb_used_data":827484,"kb_used_omap":14,"kb_used_meta":551793,"kb_avail":281736180,"statfs":{"total":289910292480,"available":288497848320,"internally_reserved":0,"allocated":847343616,"data_stored":847074706,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":565036501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8015,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8015,"num_bytes_recovered":65658880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001381"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":337257,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:17.238160+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:17.238160+0000","last_peered":"2026-02-17T15:20:17.238160+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:17.238160+0000","last_undegraded":"2026-02-17T15:20:17.238160+0000","last_fullsized":"2026-02-17T15:20:17.238160+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10712,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132766,"num_bytes_recovered":1087595092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10712,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132766,"num_bytes_recovered":1087595092,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1139187712,"data_stored":1139204096,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364834,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201310,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691888,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581788,"kb_used_data":254284,"kb_used_omap":4,"kb_used_meta":327483,"kb_avail":93790052,"statfs":{"total":96636764160,"available":96041013248,"internally_reserved":0,"allocated":260386816,"data_stored":260397190,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":335343308},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":260255744,"data_stored":260259840,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:24.559 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:24.707 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:24.707 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:24.718 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2026-02-17T15:20:23.511554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6675,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136805,"num_bytes_recovered":1120993876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1379340,"kb_used_data":827484,"kb_used_omap":14,"kb_used_meta":551793,"kb_avail":281736180,"statfs":{"total":289910292480,"available":288497848320,"internally_reserved":0,"allocated":847343616,"data_stored":847074706,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":565036501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-12052,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12052,"num_bytes_recovered":98729984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001401"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":345331,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:22.238672+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:22.238672+0000","last_peered":"2026-02-17T15:20:22.238672+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:22.238672+0000","last_undegraded":"2026-02-17T15:20:22.238672+0000","last_fullsized":"2026-02-17T15:20:22.238672+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6675,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136803,"num_bytes_recovered":1120666196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6675,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136803,"num_bytes_recovered":1120666196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1139187712,"data_stored":1139204096,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201311,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691888,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581788,"kb_used_data":254284,"kb_used_omap":4,"kb_used_meta":327483,"kb_avail":93790052,"statfs":{"total":96636764160,"available":96041013248,"internally_reserved":0,"allocated":260386816,"data_stored":260397190,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":335343308},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":260255744,"data_stored":260259840,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:24.719 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:24.863 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:24.864 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:24.874 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2026-02-17T15:20:23.511554+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6675,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136805,"num_bytes_recovered":1120993876,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1379340,"kb_used_data":827484,"kb_used_omap":14,"kb_used_meta":551793,"kb_avail":281736180,"statfs":{"total":289910292480,"available":288497848320,"internally_reserved":0,"allocated":847343616,"data_stored":847074706,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":565036501},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-12052,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12052,"num_bytes_recovered":98729984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001401"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":345331,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:22.238672+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:22.238672+0000","last_peered":"2026-02-17T15:20:22.238672+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:22.238672+0000","last_undegraded":"2026-02-17T15:20:22.238672+0000","last_fullsized":"2026-02-17T15:20:22.238672+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6675,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136803,"num_bytes_recovered":1120666196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6675,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":136803,"num_bytes_recovered":1120666196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1139187712,"data_stored":1139204096,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247913,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364835,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201311,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691888,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":581788,"kb_used_data":254284,"kb_used_omap":4,"kb_used_meta":327483,"kb_avail":93790052,"statfs":{"total":96636764160,"available":96041013248,"internally_reserved":0,"allocated":260386816,"data_stored":260397190,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":335343308},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":260255744,"data_stored":260259840,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:27.876 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:28.021 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:28.022 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:28.032 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2026-02-17T15:20:27.511998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140774,"num_bytes_recovered":1153507924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1443272,"kb_used_data":842840,"kb_used_omap":14,"kb_used_meta":600369,"kb_avail":281672248,"statfs":{"total":289910292480,"available":288432381952,"internally_reserved":0,"allocated":863068160,"data_stored":862799250,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":614778325},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-12050,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12049,"num_bytes_recovered":98705408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001378"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":353270,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:27.238454+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:27.238454+0000","last_peered":"2026-02-17T15:20:27.238454+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:27.238454+0000","last_undegraded":"2026-02-17T15:20:27.238454+0000","last_fullsized":"2026-02-17T15:20:27.238454+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140772,"num_bytes_recovered":1153180244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140772,"num_bytes_recovered":1153180244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155207168,"data_stored":1155223552,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247914,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201312,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691889,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645720,"kb_used_data":269640,"kb_used_omap":4,"kb_used_meta":376059,"kb_avail":93726120,"statfs":{"total":96636764160,"available":95975546880,"internally_reserved":0,"allocated":276111360,"data_stored":276121734,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":385085132},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":276275200,"data_stored":276279296,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:28.033 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:28.187 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:28.188 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:28.199 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2026-02-17T15:20:27.511998+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140774,"num_bytes_recovered":1153507924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1443272,"kb_used_data":842840,"kb_used_omap":14,"kb_used_meta":600369,"kb_avail":281672248,"statfs":{"total":289910292480,"available":288432381952,"internally_reserved":0,"allocated":863068160,"data_stored":862799250,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":614778325},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-12050,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12049,"num_bytes_recovered":98705408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001378"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":353270,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:27.238454+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:27.238454+0000","last_peered":"2026-02-17T15:20:27.238454+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:27.238454+0000","last_undegraded":"2026-02-17T15:20:27.238454+0000","last_fullsized":"2026-02-17T15:20:27.238454+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140772,"num_bytes_recovered":1153180244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140772,"num_bytes_recovered":1153180244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155207168,"data_stored":1155223552,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247914,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201312,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691889,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645720,"kb_used_data":269640,"kb_used_omap":4,"kb_used_meta":376059,"kb_avail":93726120,"statfs":{"total":96636764160,"available":95975546880,"internally_reserved":0,"allocated":276111360,"data_stored":276121734,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":385085132},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":276275200,"data_stored":276279296,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:31.201 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:31.351 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:31.351 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:31.360 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2026-02-17T15:20:30.587893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140774,"num_bytes_recovered":1153507924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1517900,"kb_used_data":859164,"kb_used_omap":14,"kb_used_meta":658673,"kb_avail":281597620,"statfs":{"total":289910292480,"available":288355962880,"internally_reserved":0,"allocated":879783936,"data_stored":879515026,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":674481621},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8007,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8006,"num_bytes_recovered":65585152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.076815"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":353270,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:27.238454+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:27.238454+0000","last_peered":"2026-02-17T15:20:27.238454+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:27.238454+0000","last_undegraded":"2026-02-17T15:20:27.238454+0000","last_fullsized":"2026-02-17T15:20:27.238454+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140772,"num_bytes_recovered":1153180244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140772,"num_bytes_recovered":1153180244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171910656,"data_stored":1171927040,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201312,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691890,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720348,"kb_used_data":285964,"kb_used_omap":4,"kb_used_meta":434363,"kb_avail":93651492,"statfs":{"total":96636764160,"available":95899127808,"internally_reserved":0,"allocated":292827136,"data_stored":292837510,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":444788428},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:31.361 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:31.507 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:31.507 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:31.518 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2026-02-17T15:20:30.587893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140774,"num_bytes_recovered":1153507924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1517900,"kb_used_data":859164,"kb_used_omap":14,"kb_used_meta":658673,"kb_avail":281597620,"statfs":{"total":289910292480,"available":288355962880,"internally_reserved":0,"allocated":879783936,"data_stored":879515026,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":674481621},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-8007,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8006,"num_bytes_recovered":65585152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.076815"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":353270,"reported_epoch":59,"state":"active+remapped+backfilling","last_fresh":"2026-02-17T15:20:27.238454+0000","last_change":"2026-02-17T15:20:01.771404+0000","last_active":"2026-02-17T15:20:27.238454+0000","last_peered":"2026-02-17T15:20:27.238454+0000","last_clean":"2026-02-17T15:18:48.890765+0000","last_became_active":"2026-02-17T15:18:54.280577+0000","last_became_peered":"2026-02-17T15:18:54.280577+0000","last_unstale":"2026-02-17T15:20:27.238454+0000","last_undegraded":"2026-02-17T15:20:27.238454+0000","last_fullsized":"2026-02-17T15:20:27.238454+0000","mapping_epoch":58,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140772,"num_bytes_recovered":1153180244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,2],"avail_no_missing":["3(0)","1(1)","2(2)"],"object_location_counts":[{"shards":"1(1),2(2),3(0)","objects":71529}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2705,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":140772,"num_bytes_recovered":1153180244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171910656,"data_stored":1171927040,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364836,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326544,"kb_used_data":286592,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045296,"statfs":{"total":96636764160,"available":96302383104,"internally_reserved":0,"allocated":293470208,"data_stored":293176966,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201312,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691890,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720348,"kb_used_data":285964,"kb_used_omap":4,"kb_used_meta":434363,"kb_avail":93651492,"statfs":{"total":96636764160,"available":95899127808,"internally_reserved":0,"allocated":292827136,"data_stored":292837510,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":444788428},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:34.521 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:34.675 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:34.676 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:34.688 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2026-02-17T15:20:33.512685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143479,"num_bytes_recovered":1175667284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1517912,"kb_used_data":859172,"kb_used_omap":14,"kb_used_meta":658673,"kb_avail":281597608,"statfs":{"total":289910292480,"available":288355950592,"internally_reserved":0,"allocated":879792128,"data_stored":879518486,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":674481621},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6675,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6674,"num_bytes_recovered":54673408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000901"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":358699,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-17T15:20:31.594515+0000","last_change":"2026-02-17T15:20:31.594515+0000","last_active":"2026-02-17T15:20:31.594515+0000","last_peered":"2026-02-17T15:20:31.594515+0000","last_clean":"2026-02-17T15:20:31.594515+0000","last_became_active":"2026-02-17T15:20:31.594190+0000","last_became_peered":"2026-02-17T15:20:31.594190+0000","last_unstale":"2026-02-17T15:20:31.594515+0000","last_undegraded":"2026-02-17T15:20:31.594515+0000","last_fullsized":"2026-02-17T15:20:31.594515+0000","mapping_epoch":60,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":299,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143477,"num_bytes_recovered":1175339604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143477,"num_bytes_recovered":1175339604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171910656,"data_stored":1171927040,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326556,"kb_used_data":286600,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045284,"statfs":{"total":96636764160,"available":96302370816,"internally_reserved":0,"allocated":293478400,"data_stored":293180426,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201313,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691890,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720348,"kb_used_data":285964,"kb_used_omap":4,"kb_used_meta":434363,"kb_avail":93651492,"statfs":{"total":96636764160,"available":95899127808,"internally_reserved":0,"allocated":292827136,"data_stored":292837510,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":444788428},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:34.689 INFO:tasks.ec_inconsistent_hinfo.ceph_manager:recovered! 2026-02-17T15:20:34.689 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-17T15:20:34.705 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2026-02-17T15:20:34.705 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:34.849 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:34.850 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:34.860 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2026-02-17T15:20:33.512685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143479,"num_bytes_recovered":1175667284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1517912,"kb_used_data":859172,"kb_used_omap":14,"kb_used_meta":658673,"kb_avail":281597608,"statfs":{"total":289910292480,"available":288355950592,"internally_reserved":0,"allocated":879792128,"data_stored":879518486,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":674481621},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6675,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6674,"num_bytes_recovered":54673408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000901"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":358699,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-17T15:20:31.594515+0000","last_change":"2026-02-17T15:20:31.594515+0000","last_active":"2026-02-17T15:20:31.594515+0000","last_peered":"2026-02-17T15:20:31.594515+0000","last_clean":"2026-02-17T15:20:31.594515+0000","last_became_active":"2026-02-17T15:20:31.594190+0000","last_became_peered":"2026-02-17T15:20:31.594190+0000","last_unstale":"2026-02-17T15:20:31.594515+0000","last_undegraded":"2026-02-17T15:20:31.594515+0000","last_fullsized":"2026-02-17T15:20:31.594515+0000","mapping_epoch":60,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":299,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143477,"num_bytes_recovered":1175339604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143477,"num_bytes_recovered":1175339604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171910656,"data_stored":1171927040,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326556,"kb_used_data":286600,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045284,"statfs":{"total":96636764160,"available":96302370816,"internally_reserved":0,"allocated":293478400,"data_stored":293180426,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201313,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691890,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720348,"kb_used_data":285964,"kb_used_omap":4,"kb_used_meta":434363,"kb_avail":93651492,"statfs":{"total":96636764160,"available":95899127808,"internally_reserved":0,"allocated":292827136,"data_stored":292837510,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":444788428},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:34.861 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:35.017 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:35.017 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:35.029 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2026-02-17T15:20:33.512685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143479,"num_bytes_recovered":1175667284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1517912,"kb_used_data":859172,"kb_used_omap":14,"kb_used_meta":658673,"kb_avail":281597608,"statfs":{"total":289910292480,"available":288355950592,"internally_reserved":0,"allocated":879792128,"data_stored":879518486,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":674481621},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6675,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6674,"num_bytes_recovered":54673408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000901"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":358699,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-17T15:20:31.594515+0000","last_change":"2026-02-17T15:20:31.594515+0000","last_active":"2026-02-17T15:20:31.594515+0000","last_peered":"2026-02-17T15:20:31.594515+0000","last_clean":"2026-02-17T15:20:31.594515+0000","last_became_active":"2026-02-17T15:20:31.594190+0000","last_became_peered":"2026-02-17T15:20:31.594190+0000","last_unstale":"2026-02-17T15:20:31.594515+0000","last_undegraded":"2026-02-17T15:20:31.594515+0000","last_fullsized":"2026-02-17T15:20:31.594515+0000","mapping_epoch":60,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":299,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143477,"num_bytes_recovered":1175339604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143477,"num_bytes_recovered":1175339604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171910656,"data_stored":1171927040,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326556,"kb_used_data":286600,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045284,"statfs":{"total":96636764160,"available":96302370816,"internally_reserved":0,"allocated":293478400,"data_stored":293180426,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201313,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691890,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720348,"kb_used_data":285964,"kb_used_omap":4,"kb_used_meta":434363,"kb_avail":93651492,"statfs":{"total":96636764160,"available":95899127808,"internally_reserved":0,"allocated":292827136,"data_stored":292837510,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":444788428},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:35.030 INFO:tasks.ceph.ceph_manager.ceph:clean! 2026-02-17T15:20:35.030 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:35.180 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:35.181 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:35.192 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2026-02-17T15:20:33.512685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143479,"num_bytes_recovered":1175667284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1517912,"kb_used_data":859172,"kb_used_omap":14,"kb_used_meta":658673,"kb_avail":281597608,"statfs":{"total":289910292480,"available":288355950592,"internally_reserved":0,"allocated":879792128,"data_stored":879518486,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":674481621},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6675,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6674,"num_bytes_recovered":54673408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000901"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":358699,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-17T15:20:31.594515+0000","last_change":"2026-02-17T15:20:31.594515+0000","last_active":"2026-02-17T15:20:31.594515+0000","last_peered":"2026-02-17T15:20:31.594515+0000","last_clean":"2026-02-17T15:20:31.594515+0000","last_became_active":"2026-02-17T15:20:31.594190+0000","last_became_peered":"2026-02-17T15:20:31.594190+0000","last_unstale":"2026-02-17T15:20:31.594515+0000","last_undegraded":"2026-02-17T15:20:31.594515+0000","last_fullsized":"2026-02-17T15:20:31.594515+0000","mapping_epoch":60,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":299,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143477,"num_bytes_recovered":1175339604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":49,"reported_epoch":59,"state":"active+clean","last_fresh":"2026-02-17T15:18:53.989556+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:18:53.989556+0000","last_peered":"2026-02-17T15:18:53.989556+0000","last_clean":"2026-02-17T15:18:53.989556+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:18:53.989556+0000","last_undegraded":"2026-02-17T15:18:53.989556+0000","last_fullsized":"2026-02-17T15:18:53.989556+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143477,"num_bytes_recovered":1175339604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171910656,"data_stored":1171927040,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471008,"kb_used_data":286608,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900832,"statfs":{"total":96636764160,"available":96154451968,"internally_reserved":0,"allocated":293486592,"data_stored":293500550,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326556,"kb_used_data":286600,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045284,"statfs":{"total":96636764160,"available":96302370816,"internally_reserved":0,"allocated":293478400,"data_stored":293180426,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201313,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691890,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720348,"kb_used_data":285964,"kb_used_omap":4,"kb_used_meta":434363,"kb_avail":93651492,"statfs":{"total":96636764160,"available":95899127808,"internally_reserved":0,"allocated":292827136,"data_stored":292837510,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":444788428},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:35.193 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2026-02-17T15:20:35.369 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:35.369 INFO:teuthology.orchestra.run.trial024.stdout:{"epoch":61,"fsid":"ac9bccef-327a-4812-a379-08d174e6a4bc","created":"2026-02-17T15:13:47.232501+0000","modified":"2026-02-17T15:20:31.587620+0000","last_up_change":"2026-02-17T15:18:47.882890+0000","last_in_change":"2026-02-17T15:18:51.977027+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":11,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-17T15:13:57.498851+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":4,"score_stable":4,"optimal_score":0.5,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"unique_pool_0","create_time":"2026-02-17T15:14:06.508581+0000","flags":1,"flags_names":"hashpspool","type":3,"size":3,"min_size":2,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"backfill_unfound","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"ee39a229-ca94-4bb1-ab31-dd4d57c12e53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":36,"up_from":40,"up_thru":40,"down_at":37,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6809","nonce":342414690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6810","nonce":342414690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6812","nonce":342414690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6811","nonce":342414690}]},"public_addr":"10.20.193.24:6809/342414690","cluster_addr":"10.20.193.24:6810/342414690","heartbeat_back_addr":"10.20.193.24:6812/342414690","heartbeat_front_addr":"10.20.193.24:6811/342414690","state":["exists","up"]},{"osd":1,"uuid":"85f3808e-7c1e-4d80-98f5-1929c6a71b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":10,"last_clean_end":19,"up_from":23,"up_thru":47,"down_at":20,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6805","nonce":2457068227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6806","nonce":2457068227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6808","nonce":2457068227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6807","nonce":2457068227}]},"public_addr":"10.20.193.24:6805/2457068227","cluster_addr":"10.20.193.24:6806/2457068227","heartbeat_back_addr":"10.20.193.24:6808/2457068227","heartbeat_front_addr":"10.20.193.24:6807/2457068227","state":["exists","up"]},{"osd":2,"uuid":"3e023bbc-bdaf-4440-8cc7-b8e1fbd7a3c2","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":51,"up_from":55,"up_thru":37,"down_at":52,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6813","nonce":2608472267}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6814","nonce":2608472267}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6816","nonce":2608472267}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6815","nonce":2608472267}]},"public_addr":"10.20.193.24:6813/2608472267","cluster_addr":"10.20.193.24:6814/2608472267","heartbeat_back_addr":"10.20.193.24:6816/2608472267","heartbeat_front_addr":"10.20.193.24:6815/2608472267","state":["exists","up"]},{"osd":3,"uuid":"ee3abe87-b5af-4c38-ae75-54a3f1e76cbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":30,"last_clean_end":46,"up_from":50,"up_thru":60,"down_at":47,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6801","nonce":3968139738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6802","nonce":3968139738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6804","nonce":3968139738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.24:6803","nonce":3968139738}]},"public_addr":"10.20.193.24:6801/3968139738","cluster_addr":"10.20.193.24:6802/3968139738","heartbeat_back_addr":"10.20.193.24:6804/3968139738","heartbeat_front_addr":"10.20.193.24:6803/3968139738","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.138188+0000","dead_epoch":37},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:55.488216+0000","dead_epoch":20},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2026-02-17T15:13:55.278612+0000","dead_epoch":52},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2026-02-17T15:13:54.537485+0000","dead_epoch":47}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"backfill_unfound":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-17T15:20:36.382 INFO:tasks.ceph:Scrubbing osd.0 2026-02-17T15:20:36.383 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-17T15:20:36.459 INFO:teuthology.orchestra.run.trial024.stdout:{ 2026-02-17T15:20:36.459 INFO:teuthology.orchestra.run.trial024.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-17T15:20:36.460 INFO:teuthology.orchestra.run.trial024.stdout:} 2026-02-17T15:20:36.469 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2026-02-17T15:20:36.614 INFO:teuthology.orchestra.run.trial024.stderr:instructed osd(s) 0 to deep-scrub 2026-02-17T15:20:36.625 INFO:tasks.ceph:Scrubbing osd.1 2026-02-17T15:20:36.625 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-17T15:20:36.697 INFO:teuthology.orchestra.run.trial024.stdout:{ 2026-02-17T15:20:36.697 INFO:teuthology.orchestra.run.trial024.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-17T15:20:36.697 INFO:teuthology.orchestra.run.trial024.stdout:} 2026-02-17T15:20:36.706 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2026-02-17T15:20:36.847 INFO:teuthology.orchestra.run.trial024.stderr:instructed osd(s) 1 to deep-scrub 2026-02-17T15:20:36.858 INFO:tasks.ceph:Scrubbing osd.2 2026-02-17T15:20:36.859 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-17T15:20:36.932 INFO:teuthology.orchestra.run.trial024.stdout:{ 2026-02-17T15:20:36.932 INFO:teuthology.orchestra.run.trial024.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-17T15:20:36.932 INFO:teuthology.orchestra.run.trial024.stdout:} 2026-02-17T15:20:36.941 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2026-02-17T15:20:37.085 INFO:teuthology.orchestra.run.trial024.stderr:instructed osd(s) 2 to deep-scrub 2026-02-17T15:20:37.096 INFO:tasks.ceph:Scrubbing osd.3 2026-02-17T15:20:37.097 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2026-02-17T15:20:37.172 INFO:teuthology.orchestra.run.trial024.stdout:{ 2026-02-17T15:20:37.172 INFO:teuthology.orchestra.run.trial024.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-17T15:20:37.172 INFO:teuthology.orchestra.run.trial024.stdout:} 2026-02-17T15:20:37.181 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2026-02-17T15:20:37.348 INFO:teuthology.orchestra.run.trial024.stderr:instructed osd(s) 3 to deep-scrub 2026-02-17T15:20:37.360 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:37.503 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:37.504 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:37.515 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2026-02-17T15:20:35.512925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586285866,"num_objects":71531,"num_object_clones":0,"num_object_copies":214591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71531,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":71566,"num_write_kb":572549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143479,"num_bytes_recovered":1175667284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1517936,"kb_used_data":859188,"kb_used_omap":14,"kb_used_meta":658673,"kb_avail":281597584,"statfs":{"total":289910292480,"available":288355926016,"internally_reserved":0,"allocated":879808512,"data_stored":879525385,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":674481621},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2705,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2705,"num_bytes_recovered":22159360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.000926"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":358699,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-17T15:20:31.594515+0000","last_change":"2026-02-17T15:20:31.594515+0000","last_active":"2026-02-17T15:20:31.594515+0000","last_peered":"2026-02-17T15:20:31.594515+0000","last_clean":"2026-02-17T15:20:31.594515+0000","last_became_active":"2026-02-17T15:20:31.594190+0000","last_became_peered":"2026-02-17T15:20:31.594190+0000","last_unstale":"2026-02-17T15:20:31.594515+0000","last_undegraded":"2026-02-17T15:20:31.594515+0000","last_fullsized":"2026-02-17T15:20:31.594515+0000","mapping_epoch":60,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"19'2","last_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_deep_scrub":"19'2","last_deep_scrub_stamp":"2026-02-17T15:14:39.584543+0000","last_clean_scrub_stamp":"2026-02-17T15:14:39.584543+0000","objects_scrubbed":299,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143477,"num_bytes_recovered":1175339604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"12'17","reported_seq":53,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-17T15:20:31.593500+0000","last_change":"2026-02-17T15:17:46.893867+0000","last_active":"2026-02-17T15:20:31.593500+0000","last_peered":"2026-02-17T15:20:31.593500+0000","last_clean":"2026-02-17T15:20:31.593500+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:20:31.593500+0000","last_undegraded":"2026-02-17T15:20:31.593500+0000","last_fullsized":"2026-02-17T15:20:31.593500+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-17T15:13:58.477855+0000","last_clean_scrub_stamp":"2026-02-17T15:13:58.477855+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T23:50:10.412612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585957674,"num_objects":71529,"num_object_clones":0,"num_object_copies":214587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71529,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71529,"num_write_kb":572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":143477,"num_bytes_recovered":1175339604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171910656,"data_stored":1171927040,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247916,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471032,"kb_used_data":286624,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900808,"statfs":{"total":96636764160,"available":96154427392,"internally_reserved":0,"allocated":293502976,"data_stored":293507449,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364837,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326556,"kb_used_data":286600,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045284,"statfs":{"total":96636764160,"available":96302370816,"internally_reserved":0,"allocated":293478400,"data_stored":293180426,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201313,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691890,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":720348,"kb_used_data":285964,"kb_used_omap":4,"kb_used_meta":434363,"kb_avail":93651492,"statfs":{"total":96636764160,"available":95899127808,"internally_reserved":0,"allocated":292827136,"data_stored":292837510,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":444788428},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:37.519 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-17T15:14:39.584543+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=17, tm_hour=15, tm_min=14, tm_sec=39, tm_wday=1, tm_yday=48, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=17, tm_hour=15, tm_min=20, tm_sec=35, tm_wday=1, tm_yday=48, tm_isdst=0) 2026-02-17T15:20:37.520 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-17T15:13:58.477855+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=17, tm_hour=15, tm_min=13, tm_sec=58, tm_wday=1, tm_yday=48, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=17, tm_hour=15, tm_min=20, tm_sec=35, tm_wday=1, tm_yday=48, tm_isdst=0) 2026-02-17T15:20:37.520 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-17T15:20:49.498 INFO:tasks.ceph.mgr.x.trial024.stderr:2026-02-17T15:20:49.493+0000 7f54411ee640 -1 calc_pg_upmaps abort due to max <= 0 2026-02-17T15:20:57.521 DEBUG:teuthology.orchestra.run.trial024:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2026-02-17T15:20:57.668 INFO:teuthology.orchestra.run.trial024.stdout: 2026-02-17T15:20:57.669 INFO:teuthology.orchestra.run.trial024.stderr:dumped all 2026-02-17T15:20:57.678 INFO:teuthology.orchestra.run.trial024.stdout:{"pg_ready":true,"pg_map":{"version":269,"stamp":"2026-02-17T15:20:57.515525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":586277537,"num_objects":71530,"num_object_clones":0,"num_object_copies":214588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71530,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":93,"ondisk_log_size":93,"up":5,"acting":5,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":1501968,"kb_used_data":859864,"kb_used_omap":14,"kb_used_meta":642033,"kb_avail":281613552,"statfs":{"total":289910292480,"available":288372277248,"internally_reserved":0,"allocated":880500736,"data_stored":880203371,"data_compressed":9200,"data_compressed_allocated":53248,"data_compressed_original":679936,"omap_allocated":14891,"internal_metadata":657442261},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8329,"num_objects":-1,"num_object_clones":0,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-71529,"num_write_kb":-572227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-143477,"num_bytes_recovered":-1175339604,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":-1},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001394"},"pg_stats":[{"pgid":"2.0","version":"59'71530","reported_seq":363477,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-17T15:20:49.641166+0000","last_change":"2026-02-17T15:20:49.641193+0000","last_active":"2026-02-17T15:20:49.641166+0000","last_peered":"2026-02-17T15:20:49.641166+0000","last_clean":"2026-02-17T15:20:49.641166+0000","last_became_active":"2026-02-17T15:20:31.594190+0000","last_became_peered":"2026-02-17T15:20:31.594190+0000","last_unstale":"2026-02-17T15:20:49.641166+0000","last_undegraded":"2026-02-17T15:20:49.641166+0000","last_fullsized":"2026-02-17T15:20:49.641166+0000","mapping_epoch":60,"log_start":"26'71456","ondisk_log_start":"26'71456","created":16,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'71530","last_scrub_stamp":"2026-02-17T15:20:49.641079+0000","last_deep_scrub":"59'71530","last_deep_scrub_stamp":"2026-02-17T15:20:49.641079+0000","last_clean_scrub_stamp":"2026-02-17T15:20:49.641079+0000","objects_scrubbed":71528,"log_size":74,"log_dups_size":3000,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":18,"scrub_schedule":"periodic scrub scheduled @ 2026-02-18T17:40:35.141686+0000","scrub_duration":17160,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":585949345,"num_objects":71528,"num_object_clones":0,"num_object_copies":214584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"61'19","reported_seq":64,"reported_epoch":61,"state":"active+clean","last_fresh":"2026-02-17T15:20:36.909426+0000","last_change":"2026-02-17T15:20:36.909451+0000","last_active":"2026-02-17T15:20:36.909426+0000","last_peered":"2026-02-17T15:20:36.909426+0000","last_clean":"2026-02-17T15:20:36.909426+0000","last_became_active":"2026-02-17T15:17:46.889840+0000","last_became_peered":"2026-02-17T15:17:46.889840+0000","last_unstale":"2026-02-17T15:20:36.909426+0000","last_undegraded":"2026-02-17T15:20:36.909426+0000","last_fullsized":"2026-02-17T15:20:36.909426+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'19","last_scrub_stamp":"2026-02-17T15:20:36.909386+0000","last_deep_scrub":"61'19","last_deep_scrub_stamp":"2026-02-17T15:20:36.909386+0000","last_clean_scrub_stamp":"2026-02-17T15:20:36.909386+0000","objects_scrubbed":2,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-19T02:54:32.203106+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":585949345,"num_objects":71528,"num_object_clones":0,"num_object_copies":214584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171910656,"data_stored":1171927040,"data_compressed":11466,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1696,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":3,"acting":3,"num_store_stats":4},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":328192,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":37,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":983552,"data_compressed":1779,"data_compressed_allocated":61440,"data_compressed_original":983040,"omap_allocated":0,"internal_metadata":0},"log_size":19,"ondisk_log_size":19,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":1,"up_from":23,"seq":98784247920,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":471024,"kb_used_data":286624,"kb_used_omap":4,"kb_used_meta":184379,"kb_avail":93900816,"statfs":{"total":96636764160,"available":96154435584,"internally_reserved":0,"allocated":293502976,"data_stored":293507449,"data_compressed":2723,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4394,"internal_metadata":188804822},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":1,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":50,"seq":214748364842,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":326568,"kb_used_data":286608,"kb_used_omap":5,"kb_used_meta":39930,"kb_avail":94045272,"statfs":{"total":96636764160,"available":96302358528,"internally_reserved":0,"allocated":293486592,"data_stored":293183865,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":6093,"internal_metadata":40888371},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":0,"seq":236223201318,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":40,"seq":171798691895,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":704376,"kb_used_data":286632,"kb_used_omap":4,"kb_used_meta":417723,"kb_avail":93667464,"statfs":{"total":96636764160,"available":95915483136,"internally_reserved":0,"allocated":293511168,"data_stored":293512057,"data_compressed":4045,"data_compressed_allocated":24576,"data_compressed_original":335872,"omap_allocated":4404,"internal_metadata":427749068},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":328192,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":327680,"data_compressed":593,"data_compressed_allocated":20480,"data_compressed_original":327680,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":292974592,"data_stored":292978688,"data_compressed":2130,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":3452,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":292978688,"data_stored":292982784,"data_compressed":2432,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":1696,"internal_metadata":0}]}} 2026-02-17T15:20:57.680 DEBUG:teuthology.orchestra.run.trial024:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2026-02-17T15:20:57.835 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-17T15:20:57.836 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-17T15:20:57.836 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2026-02-17T15:20:57.836 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T15:20:57.924 INFO:tasks.ceph.osd.0:Stopped 2026-02-17T15:20:57.925 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2026-02-17T15:20:57.925 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T15:20:58.023 INFO:tasks.ceph.osd.1:Stopped 2026-02-17T15:20:58.024 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2026-02-17T15:20:58.024 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T15:20:58.105 INFO:tasks.ceph.osd.2:Stopped 2026-02-17T15:20:58.106 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2026-02-17T15:20:58.106 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T15:20:59.540 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2026-02-17T15:21:04.207 INFO:tasks.ceph.osd.3:Stopped 2026-02-17T15:21:04.208 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-17T15:21:04.208 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2026-02-17T15:21:04.208 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T15:21:04.246 INFO:tasks.ceph.mgr.x:Stopped 2026-02-17T15:21:04.247 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-17T15:21:04.247 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2026-02-17T15:21:04.247 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T15:21:04.259 INFO:tasks.ceph.mon.a:Stopped 2026-02-17T15:21:04.259 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2026-02-17T15:21:04.259 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T15:21:04.313 INFO:tasks.ceph.mon.b:Stopped 2026-02-17T15:21:04.314 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2026-02-17T15:21:04.314 INFO:teuthology.orchestra.run:waiting for 300 2026-02-17T15:21:04.369 INFO:tasks.ceph.mon.c:Stopped 2026-02-17T15:21:04.370 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-17T15:21:04.370 DEBUG:teuthology.orchestra.run.trial024:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(OBJECT_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(SLOW_OPS\)' | egrep -v deep-scrub | egrep -v missing | egrep -v 'overall HEALTH_' | egrep -v repair | egrep -v 'slow request' | egrep -v unfound | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'enough copies available' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2026-02-17T15:21:04.424 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@trial024.front.sepia.ceph.com 2026-02-17T15:21:04.424 DEBUG:teuthology.orchestra.run.trial024:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2026-02-17T15:21:04.541 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2026-02-17T15:21:05.119 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@trial024.front.sepia.ceph.com 2026-02-17T15:21:05.119 DEBUG:teuthology.orchestra.run.trial024:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2026-02-17T15:21:05.140 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@trial024.front.sepia.ceph.com 2026-02-17T15:21:05.140 DEBUG:teuthology.orchestra.run.trial024:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2026-02-17T15:21:05.198 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@trial024.front.sepia.ceph.com 2026-02-17T15:21:05.198 DEBUG:teuthology.orchestra.run.trial024:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2026-02-17T15:21:05.259 INFO:tasks.ceph:Archiving mon data... 2026-02-17T15:21:05.260 DEBUG:teuthology.misc:Transferring archived files from trial024:/var/lib/ceph/mon/ceph-a to /home/teuthworker/mnt/teuthology/skanta-2026-02-17_04:10:36-rados-wip-bharath20-testing-2026-02-13-0906-squid-distro-default-trial/51568/data/mon.a.tgz 2026-02-17T15:21:05.261 DEBUG:teuthology.orchestra.run.trial024:> mktemp 2026-02-17T15:21:05.305 INFO:teuthology.orchestra.run.trial024.stdout:/tmp/tmp.0kvR1VmCJ6 2026-02-17T15:21:05.305 DEBUG:teuthology.orchestra.run.trial024:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.0kvR1VmCJ6 2026-02-17T15:21:05.389 DEBUG:teuthology.orchestra.run.trial024:> sudo chmod 0666 /tmp/tmp.0kvR1VmCJ6 2026-02-17T15:21:05.487 DEBUG:teuthology.orchestra.remote:trial024:/tmp/tmp.0kvR1VmCJ6 is 291KB 2026-02-17T15:21:05.543 DEBUG:teuthology.orchestra.run.trial024:> rm -fr /tmp/tmp.0kvR1VmCJ6 2026-02-17T15:21:05.547 DEBUG:teuthology.misc:Transferring archived files from trial024:/var/lib/ceph/mon/ceph-b to /home/teuthworker/mnt/teuthology/skanta-2026-02-17_04:10:36-rados-wip-bharath20-testing-2026-02-13-0906-squid-distro-default-trial/51568/data/mon.b.tgz 2026-02-17T15:21:05.548 DEBUG:teuthology.orchestra.run.trial024:> mktemp 2026-02-17T15:21:05.593 INFO:teuthology.orchestra.run.trial024.stdout:/tmp/tmp.emiaSrOYvR 2026-02-17T15:21:05.593 DEBUG:teuthology.orchestra.run.trial024:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.emiaSrOYvR 2026-02-17T15:21:05.687 DEBUG:teuthology.orchestra.run.trial024:> sudo chmod 0666 /tmp/tmp.emiaSrOYvR 2026-02-17T15:21:05.747 DEBUG:teuthology.orchestra.remote:trial024:/tmp/tmp.emiaSrOYvR is 373KB 2026-02-17T15:21:05.803 DEBUG:teuthology.orchestra.run.trial024:> rm -fr /tmp/tmp.emiaSrOYvR 2026-02-17T15:21:05.807 DEBUG:teuthology.misc:Transferring archived files from trial024:/var/lib/ceph/mon/ceph-c to /home/teuthworker/mnt/teuthology/skanta-2026-02-17_04:10:36-rados-wip-bharath20-testing-2026-02-13-0906-squid-distro-default-trial/51568/data/mon.c.tgz 2026-02-17T15:21:05.807 DEBUG:teuthology.orchestra.run.trial024:> mktemp 2026-02-17T15:21:05.853 INFO:teuthology.orchestra.run.trial024.stdout:/tmp/tmp.kEk5dDTETh 2026-02-17T15:21:05.854 DEBUG:teuthology.orchestra.run.trial024:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.kEk5dDTETh 2026-02-17T15:21:05.949 DEBUG:teuthology.orchestra.run.trial024:> sudo chmod 0666 /tmp/tmp.kEk5dDTETh 2026-02-17T15:21:06.003 DEBUG:teuthology.orchestra.remote:trial024:/tmp/tmp.kEk5dDTETh is 373KB 2026-02-17T15:21:06.059 DEBUG:teuthology.orchestra.run.trial024:> rm -fr /tmp/tmp.kEk5dDTETh 2026-02-17T15:21:06.063 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-17T15:21:06.064 DEBUG:teuthology.orchestra.run.trial024:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2026-02-17T15:21:06.157 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-17T15:21:06.157 INFO:tasks.ceph:Archiving crash dumps... 2026-02-17T15:21:06.160 DEBUG:teuthology.misc:Transferring archived files from trial024:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/skanta-2026-02-17_04:10:36-rados-wip-bharath20-testing-2026-02-13-0906-squid-distro-default-trial/51568/remote/trial024/crash 2026-02-17T15:21:06.161 DEBUG:teuthology.orchestra.run.trial024:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-17T15:21:06.205 INFO:tasks.ceph:Compressing logs... 2026-02-17T15:21:06.205 DEBUG:teuthology.orchestra.run.trial024:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-17T15:21:06.258 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43188.log 2026-02-17T15:21:06.261 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32388.log 2026-02-17T15:21:06.261 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30772.log 2026-02-17T15:21:06.261 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43188.log.gz 2026-02-17T15:21:06.261 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36308.log 2026-02-17T15:21:06.261 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32388.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32388.log.gz 2026-02-17T15:21:06.261 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44387.log 2026-02-17T15:21:06.261 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30772.log.gz 2026-02-17T15:21:06.261 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.36308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36308.log.gz 2026-02-17T15:21:06.261 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43723.log 2026-02-17T15:21:06.262 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43488.log 2026-02-17T15:21:06.262 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44387.log: /var/log/ceph/ceph-client.admin.43723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44387.log.gz 2026-02-17T15:21:06.262 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43723.log.gz 2026-02-17T15:21:06.262 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44174.log 2026-02-17T15:21:06.262 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45234.log 2026-02-17T15:21:06.262 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43488.log.gz 2026-02-17T15:21:06.262 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31084.log 2026-02-17T15:21:06.262 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44174.log.gz 2026-02-17T15:21:06.285 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34751.log 2026-02-17T15:21:06.285 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45234.log.gz 2026-02-17T15:21:06.285 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31869.log 2026-02-17T15:21:06.285 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31084.log.gz 2026-02-17T15:21:06.285 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34751.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34751.log.gz 2026-02-17T15:21:06.285 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42638.log 2026-02-17T15:21:06.286 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42406.log 2026-02-17T15:21:06.286 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31869.log.gz 2026-02-17T15:21:06.286 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23802.log 2026-02-17T15:21:06.286 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42638.log: /var/log/ceph/ceph-client.admin.42406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42638.log.gz 2026-02-17T15:21:06.286 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42406.log.gz 2026-02-17T15:21:06.286 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20356.log 2026-02-17T15:21:06.286 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.23802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27418.log 2026-02-17T15:21:06.286 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23802.log.gz 2026-02-17T15:21:06.286 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35665.log 2026-02-17T15:21:06.286 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20356.log.gz 2026-02-17T15:21:06.286 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24134.log 2026-02-17T15:21:06.286 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27418.log.gz 2026-02-17T15:21:06.287 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35665.log.gz 2026-02-17T15:21:06.287 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2026-02-17T15:21:06.287 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40634.log 2026-02-17T15:21:06.287 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24134.log: /var/log/ceph/ceph-mon.b.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24134.log.gz 2026-02-17T15:21:06.287 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29646.log 2026-02-17T15:21:06.287 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40634.log.gz 2026-02-17T15:21:06.287 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40739.log 2026-02-17T15:21:06.287 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30973.log 2026-02-17T15:21:06.287 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29646.log.gz 2026-02-17T15:21:06.287 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40739.log.gz 2026-02-17T15:21:06.287 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44765.log 2026-02-17T15:21:06.287 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28172.log 2026-02-17T15:21:06.287 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30973.log.gz 2026-02-17T15:21:06.288 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20770.log 2026-02-17T15:21:06.288 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44765.log.gz 2026-02-17T15:21:06.288 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28616.log 2026-02-17T15:21:06.288 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28172.log.gz 2026-02-17T15:21:06.288 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20770.log.gz 2026-02-17T15:21:06.288 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40274.log 2026-02-17T15:21:06.288 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28616.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.11835.log 2026-02-17T15:21:06.288 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28616.log.gz 2026-02-17T15:21:06.288 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31217.log 2026-02-17T15:21:06.288 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40274.log.gz 2026-02-17T15:21:06.288 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40406.log 2026-02-17T15:21:06.288 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph.tmp-client.admin.11835.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.11835.log.gz 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21568.log 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31217.log.gz 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42510.log 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40406.log.gz 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr:gzip/var/log/ceph/ceph-client.admin.21568.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42063.log 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21568.log.gz 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34656.log 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42510.log.gz 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43941.log 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42063.log.gz 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32387.log 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34656.log.gz 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21467.log 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43941.log.gz 2026-02-17T15:21:06.289 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.32387.log: -- /var/log/ceph/ceph-client.admin.24371.log 2026-02-17T15:21:06.290 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32387.log.gz 2026-02-17T15:21:06.290 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27521.log 2026-02-17T15:21:06.290 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21467.log.gz 2026-02-17T15:21:06.290 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26938.log 2026-02-17T15:21:06.290 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24371.log.gz 2026-02-17T15:21:06.290 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42744.log 2026-02-17T15:21:06.290 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27521.log: /var/log/ceph/ceph-client.admin.26938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27521.log.gz 2026-02-17T15:21:06.290 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26938.log.gz 2026-02-17T15:21:06.290 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32351.log 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28407.log 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42744.log.gz 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43840.log 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32351.log.gz 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32495.log 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28407.log: /var/log/ceph/ceph-client.admin.43840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28407.log.gz 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43840.log.gz 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35927.log 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28840.log 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32495.log: /var/log/ceph/ceph-client.admin.35927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32495.log.gz 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35927.log.gz 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28299.log 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21491.log 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28840.log.gz 2026-02-17T15:21:06.291 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42064.log 2026-02-17T15:21:06.292 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28299.log.gz 2026-02-17T15:21:06.292 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35222.log 2026-02-17T15:21:06.292 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21491.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21491.log.gz 2026-02-17T15:21:06.292 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42064.log.gz 2026-02-17T15:21:06.292 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35294.log 2026-02-17T15:21:06.292 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43832.log 2026-02-17T15:21:06.292 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35222.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35222.log.gz 2026-02-17T15:21:06.292 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23998.log 2026-02-17T15:21:06.292 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35294.log: /var/log/ceph/ceph-client.admin.43832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35294.log.gz 2026-02-17T15:21:06.292 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43832.log.gz 2026-02-17T15:21:06.292 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30432.log 2026-02-17T15:21:06.292 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35532.log 2026-02-17T15:21:06.292 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.23998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23998.log.gz 2026-02-17T15:21:06.292 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27077.log 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30432.log.gz 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35892.log 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35532.log.gz 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45416.log 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27077.log.gz 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30307.log 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35892.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35892.log.gz 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27417.log 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45416.log.gz 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30307.log.gz 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24269.log 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29084.log 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27417.log.gz 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27415.log 2026-02-17T15:21:06.293 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24269.log: /var/log/ceph/ceph-client.admin.29084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44497.log 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24269.log.gz 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29084.log.gz 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27415.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27415.log.gz 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24030.log 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43521.log 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44497.log.gz 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr:gzip/var/log/ceph/ceph-client.admin.24030.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.40168.log 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24030.log.gz 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32206.log 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43521.log.gz 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42842.log 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40168.log.gz 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32101.log 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32206.log.gz 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35531.log 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42842.log.gz 2026-02-17T15:21:06.294 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29278.log 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32101.log.gz 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34856.log 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35531.log.gz 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44386.log 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29278.log.gz 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34856.log.gz 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45364.log 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45286.log 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44386.log.gz 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24362.log 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45364.log.gz 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23898.log 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45286.log.gz 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35928.log 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24362.log: /var/log/ceph/ceph-client.admin.23898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24362.log.gz 2026-02-17T15:21:06.295 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23898.log.gz 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42407.log 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30060.log 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35928.log: /var/log/ceph/ceph-client.admin.42407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32065.log 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42407.log.gz 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35928.log.gz 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30060.log.gz 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32205.log 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43627.log 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32065.log.gz 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34993.log 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32205.log.gz 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43627.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43627.log.gz 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30539.log 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24674.log 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34993.log.gz 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44609.log 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30539.log: /var/log/ceph/ceph-client.admin.24674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30539.log.gz 2026-02-17T15:21:06.296 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24674.log.gz 2026-02-17T15:21:06.297 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35434.log 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41486.log 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44609.log: /var/log/ceph/ceph-client.admin.35434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35434.log.gz 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27718.log 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44609.log.gz 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29673.log 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41486.log: /var/log/ceph/ceph-client.admin.27718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41486.log.gz 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27718.log.gz 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26574.log 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29673.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29673.log.gz 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45104.log 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31418.log 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26574.log.gz 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.45104.log: -- /var/log/ceph/ceph-client.admin.31970.log 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45104.log.gz 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28264.log 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31418.log.gz 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21772.log 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31970.log.gz 2026-02-17T15:21:06.312 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44973.log 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28264.log.gz 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28067.log 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21772.log: /var/log/ceph/ceph-client.admin.44973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42174.log 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44973.log.gz 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21772.log.gz 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28067.log.gz 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28302.log 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32102.log 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21442.log 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42174.log.gz 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28302.log: /var/log/ceph/ceph-client.admin.32102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28302.log.gz 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27722.log 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32102.log.gz 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21442.log.gz 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42277.log 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.27722.log: -- /var/log/ceph/ceph-client.admin.30644.log 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27722.log.gz 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42277.log.gz 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31661.log 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.30644.log: -- /var/log/ceph/ceph-client.admin.30870.log 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30644.log.gz 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31661.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31661.log.gz 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40172.log 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24602.log 2026-02-17T15:21:06.313 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30870.log.gz 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40172.log.gz 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24602.log.gz 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30398.log 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41617.log 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27308.log 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30398.log.gz 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31562.log 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41617.log.gz 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21047.log 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27308.log.gz 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43833.log 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31562.log.gz 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21047.log.gz 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35136.log 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45312.log 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43833.log.gz 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27416.log 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35136.log.gz 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34654.log 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45312.log.gz 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27416.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27416.log.gz 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42975.log 2026-02-17T15:21:06.314 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26730.log 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34654.log.gz 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42975.log.gz 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44389.log 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27822.log 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26730.log.gz 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44389.log.gz 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26739.log 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27822.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27822.log.gz 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28398.log 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42743.log 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26739.log.gz 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44661.log 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28398.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.28398.log.gz -5 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr: --verbose -- /var/log/ceph/ceph-client.admin.27963.log 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42743.log.gz 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24005.log 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44661.log.gz 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43630.log 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27963.log.gz 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44817.log 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24005.log.gz 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21100.log 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43630.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43630.log.gz 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44817.log.gz 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43968.log 2026-02-17T15:21:06.315 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44511.log 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21100.log.gz 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26742.log 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43968.log: /var/log/ceph/ceph-client.admin.44511.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43968.log.gz 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44511.log.gz 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27952.log 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42056.log 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26742.log: /var/log/ceph/ceph-client.admin.27952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27952.log.gz 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26742.log.gz 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43284.log 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23799.log/var/log/ceph/ceph-client.admin.42056.log: 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42056.log.gz 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43284.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30540.log 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43284.log.gz 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.23799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23799.log.gz 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20941.log 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30540.log.gz 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29512.log 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43185.log 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20941.log.gz 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24478.log 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29512.log.gz 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43185.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.31085.log -- replaced with /var/log/ceph/ceph-client.admin.43185.log.gz 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr: 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31973.log 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24478.log.gz 2026-02-17T15:21:06.316 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26738.log 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31085.log.gz 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31973.log.gz 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41836.log 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26738.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26738.log.gz 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27520.log 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29956.log 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41836.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41836.log.gz 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27520.log.gz 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29517.log 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41186.log 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29956.log.gz 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45156.log 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29517.log.gz 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41186.log.gz 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28849.log 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20664.log 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45156.log.gz 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29544.log 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28849.log.gz 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20664.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.20664.log.gz --verbose 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr: -- /var/log/ceph/ceph-client.admin.41694.log 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29544.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29544.log.gz 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30199.log 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41694.log.gz 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36034.log 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26976.log 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30199.log.gz 2026-02-17T15:21:06.317 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.36034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36034.log.gz 2026-02-17T15:21:06.318 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32767.log 2026-02-17T15:21:06.318 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26976.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41383.log 2026-02-17T15:21:06.318 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26976.log.gz 2026-02-17T15:21:06.318 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32767.log.gz 2026-02-17T15:21:06.318 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43186.log 2026-02-17T15:21:06.318 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28841.log 2026-02-17T15:21:06.318 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41383.log.gz 2026-02-17T15:21:06.318 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20889.log 2026-02-17T15:21:06.318 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43186.log.gz 2026-02-17T15:21:06.318 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29989.log 2026-02-17T15:21:06.318 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28841.log.gz 2026-02-17T15:21:06.318 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44089.log 2026-02-17T15:21:06.318 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20889.log.gz 2026-02-17T15:21:06.318 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29989.log.gz 2026-02-17T15:21:06.321 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41290.log 2026-02-17T15:21:06.321 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44089.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.44089.log.gz 2026-02-17T15:21:06.321 INFO:teuthology.orchestra.run.trial024.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42136.log 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41290.log.gz 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40405.log 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5/var/log/ceph/ceph-client.admin.42136.log: --verbose -- /var/log/ceph/ceph-client.admin.41153.log 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42136.log.gz 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40405.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.29760.log -- replaced with /var/log/ceph/ceph-client.admin.40405.log.gz 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr: 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35930.log 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41153.log.gz 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29760.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29760.log.gz 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35930.log.gz 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28975.log 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose/var/log/ceph/ceph-mgr.x.log: -- /var/log/ceph/ceph-client.admin.30867.log 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28975.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28975.log.gz 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38156.log 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42172.log 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30867.log.gz 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.38156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38156.log.gz 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44178.log 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27080.log 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42172.log.gz 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45208.log 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44178.log.gz 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:gzip/var/log/ceph/ceph-client.admin.27080.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34884.log 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27080.log.gz 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45208.log.gz 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28635.log 2026-02-17T15:21:06.322 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21672.log 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34884.log.gz 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28635.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28635.log.gz 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.21671.log 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21672.log.gz 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42604.log 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr:gzip/var/log/ceph/ceph-client.admin.21671.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30085.log 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21671.log.gz 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40512.log 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42604.log.gz 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24476.log 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30085.log.gz 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26637.log 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40512.log.gz 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26600.log 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24476.log.gz 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42947.log 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26637.log.gz 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26600.log.gz 2026-02-17T15:21:06.323 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42171.log 2026-02-17T15:21:06.324 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40602.log 2026-02-17T15:21:06.324 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42947.log.gz 2026-02-17T15:21:06.324 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42171.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42171.log.gz 2026-02-17T15:21:06.324 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27380.log 2026-02-17T15:21:06.324 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20967.log 2026-02-17T15:21:06.324 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40602.log.gz 2026-02-17T15:21:06.324 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27380.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27380.log.gz 2026-02-17T15:21:06.324 INFO:teuthology.orchestra.run.trial024.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43734.log 2026-02-17T15:21:06.324 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24466.log 2026-02-17T15:21:06.324 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20967.log.gz 2026-02-17T15:21:06.324 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24108.log 2026-02-17T15:21:06.325 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43734.log.gz 2026-02-17T15:21:06.325 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23764.log 2026-02-17T15:21:06.325 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24466.log.gz 2026-02-17T15:21:06.325 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24108.log.gz 2026-02-17T15:21:06.325 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35190.log 2026-02-17T15:21:06.325 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5/var/log/ceph/ceph-client.admin.23764.log: --verbose -- /var/log/ceph/ceph-client.admin.44999.log 2026-02-17T15:21:06.325 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23764.log.gz 2026-02-17T15:21:06.325 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35190.log.gz 2026-02-17T15:21:06.325 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24697.log 2026-02-17T15:21:06.326 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44999.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44999.log.gz 2026-02-17T15:21:06.326 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27748.log 2026-02-17T15:21:06.326 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45338.log 2026-02-17T15:21:06.326 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24697.log.gz 2026-02-17T15:21:06.326 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5/var/log/ceph/ceph-client.admin.27748.log: --verbose -- /var/log/ceph/ceph-client.admin.43940.log 2026-02-17T15:21:06.326 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27748.log.gz 2026-02-17T15:21:06.327 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45338.log.gz 2026-02-17T15:21:06.327 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28070.log 2026-02-17T15:21:06.327 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42032.log 2026-02-17T15:21:06.327 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43940.log.gz 2026-02-17T15:21:06.327 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21924.log 2026-02-17T15:21:06.327 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28070.log.gz 2026-02-17T15:21:06.328 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32325.log 2026-02-17T15:21:06.328 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42032.log.gz 2026-02-17T15:21:06.328 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21924.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21924.log.gz 2026-02-17T15:21:06.328 INFO:teuthology.orchestra.run.trial024.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42708.log 2026-02-17T15:21:06.328 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32325.log.gz 2026-02-17T15:21:06.328 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41393.log 2026-02-17T15:21:06.329 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29316.log 2026-02-17T15:21:06.329 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42708.log.gz 2026-02-17T15:21:06.329 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21438.log 2026-02-17T15:21:06.329 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41393.log.gz 2026-02-17T15:21:06.330 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28741.log 2026-02-17T15:21:06.330 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29316.log.gz 2026-02-17T15:21:06.330 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21438.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21438.log.gz 2026-02-17T15:21:06.330 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43628.log 2026-02-17T15:21:06.330 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30771.log 2026-02-17T15:21:06.330 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28741.log.gz 2026-02-17T15:21:06.331 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31972.log 2026-02-17T15:21:06.331 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43628.log.gz 2026-02-17T15:21:06.331 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26974.log 2026-02-17T15:21:06.331 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30771.log.gz 2026-02-17T15:21:06.331 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44285.log 2026-02-17T15:21:06.331 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31972.log.gz 2026-02-17T15:21:06.332 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21386.log 2026-02-17T15:21:06.332 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26974.log.gz 2026-02-17T15:21:06.332 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21074.log 2026-02-17T15:21:06.332 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44285.log.gz 2026-02-17T15:21:06.333 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43384.log 2026-02-17T15:21:06.333 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21386.log.gz 2026-02-17T15:21:06.333 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35302.log 2026-02-17T15:21:06.333 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21074.log.gz 2026-02-17T15:21:06.333 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34992.log 2026-02-17T15:21:06.333 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43384.log.gz 2026-02-17T15:21:06.334 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35093.log 2026-02-17T15:21:06.334 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35302.log.gz 2026-02-17T15:21:06.334 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42408.log 2026-02-17T15:21:06.334 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34992.log.gz 2026-02-17T15:21:06.335 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29759.log 2026-02-17T15:21:06.335 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35093.log.gz 2026-02-17T15:21:06.335 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42408.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42408.log.gz 2026-02-17T15:21:06.335 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40498.log 2026-02-17T15:21:06.335 INFO:teuthology.orchestra.run.trial024.stderr:gzip/var/log/ceph/ceph-client.admin.29759.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35840.log 2026-02-17T15:21:06.336 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29759.log.gz 2026-02-17T15:21:06.336 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40498.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40498.log.gz 2026-02-17T15:21:06.336 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26636.log 2026-02-17T15:21:06.336 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38182.log 2026-02-17T15:21:06.336 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35840.log.gz 2026-02-17T15:21:06.336 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44037.log 2026-02-17T15:21:06.337 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26636.log.gz 2026-02-17T15:21:06.337 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2026-02-17T15:21:06.337 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.38182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38182.log.gz 2026-02-17T15:21:06.337 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44282.log 2026-02-17T15:21:06.337 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44037.log.gz 2026-02-17T15:21:06.338 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45051.log 2026-02-17T15:21:06.338 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35632.log 2026-02-17T15:21:06.338 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44282.log.gz 2026-02-17T15:21:06.338 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28397.log 2026-02-17T15:21:06.338 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45051.log.gz 2026-02-17T15:21:06.339 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43187.log 2026-02-17T15:21:06.339 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35632.log.gz 2026-02-17T15:21:06.339 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31765.log 2026-02-17T15:21:06.339 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28397.log.gz 2026-02-17T15:21:06.339 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36129.log 2026-02-17T15:21:06.340 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43187.log.gz 2026-02-17T15:21:06.340 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41837.log 2026-02-17T15:21:06.340 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31765.log.gz 2026-02-17T15:21:06.340 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.36129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36129.log.gz 2026-02-17T15:21:06.340 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40738.log 2026-02-17T15:21:06.341 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16819.log 2026-02-17T15:21:06.341 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41837.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41837.log.gz 2026-02-17T15:21:06.341 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40738.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40738.log.gz 2026-02-17T15:21:06.341 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20473.log 2026-02-17T15:21:06.341 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24232.log 2026-02-17T15:21:06.341 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.16819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16819.log.gz 2026-02-17T15:21:06.342 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20473.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20473.log.gz 2026-02-17T15:21:06.342 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44490.log 2026-02-17T15:21:06.342 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24232.log.gz 2026-02-17T15:21:06.342 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31314.log 2026-02-17T15:21:06.342 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24570.log 2026-02-17T15:21:06.343 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44490.log.gz 2026-02-17T15:21:06.343 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31314.log.gz 2026-02-17T15:21:06.343 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30200.log 2026-02-17T15:21:06.343 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29177.log 2026-02-17T15:21:06.343 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24570.log.gz 2026-02-17T15:21:06.344 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26972.log 2026-02-17T15:21:06.344 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30200.log.gz 2026-02-17T15:21:06.344 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26844.log 2026-02-17T15:21:06.344 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29177.log.gz 2026-02-17T15:21:06.344 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26972.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26972.log.gz 2026-02-17T15:21:06.344 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40275.log 2026-02-17T15:21:06.345 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27176.log 2026-02-17T15:21:06.345 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26844.log.gz 2026-02-17T15:21:06.345 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28530.log 2026-02-17T15:21:06.345 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40275.log.gz 2026-02-17T15:21:06.345 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27176.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.21338.log -- replaced with /var/log/ceph/ceph-client.admin.27176.log.gz 2026-02-17T15:21:06.345 INFO:teuthology.orchestra.run.trial024.stderr: 2026-02-17T15:21:06.346 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43079.log 2026-02-17T15:21:06.346 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28530.log.gz 2026-02-17T15:21:06.346 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21338.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21338.log.gz 2026-02-17T15:21:06.346 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41148.log 2026-02-17T15:21:06.346 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21723.log 2026-02-17T15:21:06.346 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43079.log.gz 2026-02-17T15:21:06.347 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41148.log.gz 2026-02-17T15:21:06.347 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27280.log 2026-02-17T15:21:06.347 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31764.log 2026-02-17T15:21:06.347 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21723.log.gz 2026-02-17T15:21:06.347 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27280.log.gz 2026-02-17T15:21:06.347 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28602.log 2026-02-17T15:21:06.348 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28744.log 2026-02-17T15:21:06.348 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31764.log.gz 2026-02-17T15:21:06.348 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27078.log 2026-02-17T15:21:06.348 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28602.log.gz 2026-02-17T15:21:06.349 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29758.log 2026-02-17T15:21:06.349 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28744.log.gz 2026-02-17T15:21:06.349 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27078.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.27078.log.gz 2026-02-17T15:21:06.349 INFO:teuthology.orchestra.run.trial024.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35544.log 2026-02-17T15:21:06.349 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29758.log.gz 2026-02-17T15:21:06.349 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24006.log 2026-02-17T15:21:06.350 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31322.log 2026-02-17T15:21:06.350 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35544.log.gz 2026-02-17T15:21:06.350 INFO:teuthology.orchestra.run.trial024.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2026-02-17T15:21:06.350 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34956.log 2026-02-17T15:21:06.350 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24006.log.gz 2026-02-17T15:21:06.350 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41730.log 2026-02-17T15:21:06.351 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31322.log.gz 2026-02-17T15:21:06.351 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23905.log 2026-02-17T15:21:06.351 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34956.log.gz 2026-02-17T15:21:06.351 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41730.log: /var/log/ceph/ceph-client.admin.20562.log 2026-02-17T15:21:06.351 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41730.log.gz 2026-02-17T15:21:06.351 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.23905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23905.log.gz 2026-02-17T15:21:06.352 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32195.log 2026-02-17T15:21:06.352 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24730.log 2026-02-17T15:21:06.352 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20562.log: /var/log/ceph/ceph-client.admin.32195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20562.log.gz 2026-02-17T15:21:06.352 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41835.log 2026-02-17T15:21:06.352 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32195.log.gz 2026-02-17T15:21:06.353 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32485.log 2026-02-17T15:21:06.353 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24730.log: /var/log/ceph/ceph-client.admin.41835.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24730.log.gz 2026-02-17T15:21:06.353 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41835.log.gz 2026-02-17T15:21:06.353 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40706.log 2026-02-17T15:21:06.353 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32485.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32485.log.gz 2026-02-17T15:21:06.353 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43416.log 2026-02-17T15:21:06.353 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16891.log 2026-02-17T15:21:06.353 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40706.log.gz 2026-02-17T15:21:06.354 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27859.log 2026-02-17T15:21:06.354 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43416.log.gz 2026-02-17T15:21:06.354 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.16891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41824.log 2026-02-17T15:21:06.354 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16891.log.gz 2026-02-17T15:21:06.354 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24268.log 2026-02-17T15:21:06.355 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27859.log.gz 2026-02-17T15:21:06.355 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44177.log 2026-02-17T15:21:06.355 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41824.log.gz 2026-02-17T15:21:06.355 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24268.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40810.log 2026-02-17T15:21:06.355 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24268.log.gz 2026-02-17T15:21:06.355 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27519.log 2026-02-17T15:21:06.356 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44177.log.gz 2026-02-17T15:21:06.356 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42942.log 2026-02-17T15:21:06.356 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40810.log.gz 2026-02-17T15:21:06.356 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42746.log 2026-02-17T15:21:06.356 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27519.log.gz 2026-02-17T15:21:06.356 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28056.log 2026-02-17T15:21:06.357 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42942.log: /var/log/ceph/ceph-client.admin.42746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42942.log.gz 2026-02-17T15:21:06.357 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42746.log.gz 2026-02-17T15:21:06.357 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31527.log 2026-02-17T15:21:06.357 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34883.log 2026-02-17T15:21:06.357 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28056.log.gz 2026-02-17T15:21:06.357 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43046.log 2026-02-17T15:21:06.358 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31527.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31527.log.gz 2026-02-17T15:21:06.358 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32484.log 2026-02-17T15:21:06.358 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34883.log.gz 2026-02-17T15:21:06.358 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43046.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43046.log.gz 2026-02-17T15:21:06.358 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43150.log 2026-02-17T15:21:06.359 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24578.log 2026-02-17T15:21:06.359 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32484.log.gz 2026-02-17T15:21:06.359 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40067.log 2026-02-17T15:21:06.359 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43150.log.gz 2026-02-17T15:21:06.359 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24578.log.gz 2026-02-17T15:21:06.359 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23801.log 2026-02-17T15:21:06.359 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44687.log 2026-02-17T15:21:06.360 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40067.log.gz 2026-02-17T15:21:06.360 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.23801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23801.log.gz 2026-02-17T15:21:06.360 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31762.log 2026-02-17T15:21:06.360 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40846.log 2026-02-17T15:21:06.360 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44687.log.gz 2026-02-17T15:21:06.361 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40276.log 2026-02-17T15:21:06.361 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31762.log.gz 2026-02-17T15:21:06.361 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42841.log 2026-02-17T15:21:06.361 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40846.log.gz 2026-02-17T15:21:06.361 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40276.log.gz 2026-02-17T15:21:06.361 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41202.log 2026-02-17T15:21:06.362 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16930.log 2026-02-17T15:21:06.362 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42841.log.gz 2026-02-17T15:21:06.362 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31868.log 2026-02-17T15:21:06.362 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41202.log.gz 2026-02-17T15:21:06.362 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.16930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41494.log 2026-02-17T15:21:06.362 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16930.log.gz 2026-02-17T15:21:06.363 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35197.log 2026-02-17T15:21:06.363 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31868.log.gz 2026-02-17T15:21:06.363 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28743.log 2026-02-17T15:21:06.363 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41494.log.gz 2026-02-17T15:21:06.364 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27857.log 2026-02-17T15:21:06.364 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35197.log.gz 2026-02-17T15:21:06.364 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35740.log 2026-02-17T15:21:06.364 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28743.log.gz 2026-02-17T15:21:06.364 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27857.log.gz 2026-02-17T15:21:06.364 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20769.log 2026-02-17T15:21:06.365 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40510.log 2026-02-17T15:21:06.365 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35740.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35740.log.gz 2026-02-17T15:21:06.365 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20769.log.gz 2026-02-17T15:21:06.365 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41729.log 2026-02-17T15:21:06.365 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40510.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40510.log.gz 2026-02-17T15:21:06.365 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34774.log 2026-02-17T15:21:06.366 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29757.log 2026-02-17T15:21:06.366 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41729.log.gz 2026-02-17T15:21:06.366 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34774.log.gz 2026-02-17T15:21:06.366 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26973.log 2026-02-17T15:21:06.366 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44063.log 2026-02-17T15:21:06.367 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29757.log.gz 2026-02-17T15:21:06.367 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26973.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26973.log.gz 2026-02-17T15:21:06.367 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32494.log 2026-02-17T15:21:06.367 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41289.log 2026-02-17T15:21:06.367 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44063.log.gz 2026-02-17T15:21:06.368 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27965.log 2026-02-17T15:21:06.368 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32494.log.gz 2026-02-17T15:21:06.368 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34655.log 2026-02-17T15:21:06.368 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41289.log.gz 2026-02-17T15:21:06.368 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27965.log.gz 2026-02-17T15:21:06.368 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43726.log 2026-02-17T15:21:06.369 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36282.log 2026-02-17T15:21:06.369 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34655.log.gz 2026-02-17T15:21:06.369 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32299.log 2026-02-17T15:21:06.369 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43726.log.gz 2026-02-17T15:21:06.369 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.36282.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36282.log.gz 2026-02-17T15:21:06.369 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27276.log 2026-02-17T15:21:06.370 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24270.log 2026-02-17T15:21:06.370 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32299.log.gz 2026-02-17T15:21:06.370 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31077.log 2026-02-17T15:21:06.370 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27276.log.gz 2026-02-17T15:21:06.370 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24270.log.gz 2026-02-17T15:21:06.371 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21019.log 2026-02-17T15:21:06.371 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29865.log 2026-02-17T15:21:06.371 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31077.log.gz 2026-02-17T15:21:06.371 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41960.log 2026-02-17T15:21:06.371 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21019.log.gz 2026-02-17T15:21:06.372 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44583.log 2026-02-17T15:21:06.372 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29865.log.gz 2026-02-17T15:21:06.372 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28170.log 2026-02-17T15:21:06.372 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41960.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41960.log.gz 2026-02-17T15:21:06.372 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44583.log.gz 2026-02-17T15:21:06.372 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35768.log 2026-02-17T15:21:06.373 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42850.log 2026-02-17T15:21:06.373 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28170.log.gz 2026-02-17T15:21:06.373 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35768.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41044.log 2026-02-17T15:21:06.373 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35768.log.gz 2026-02-17T15:21:06.374 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42850.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42850.log.gz 2026-02-17T15:21:06.374 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31423.log 2026-02-17T15:21:06.374 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41731.log 2026-02-17T15:21:06.374 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41044.log.gz 2026-02-17T15:21:06.374 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41928.log 2026-02-17T15:21:06.374 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31423.log.gz 2026-02-17T15:21:06.375 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41394.log 2026-02-17T15:21:06.375 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41731.log.gz 2026-02-17T15:21:06.375 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21724.log 2026-02-17T15:21:06.375 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41928.log.gz 2026-02-17T15:21:06.376 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45443.log 2026-02-17T15:21:06.376 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41394.log.gz 2026-02-17T15:21:06.376 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29313.log 2026-02-17T15:21:06.376 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21724.log.gz 2026-02-17T15:21:06.376 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45443.log.gz 2026-02-17T15:21:06.376 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32585.log 2026-02-17T15:21:06.377 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35666.log 2026-02-17T15:21:06.377 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29313.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29313.log.gz 2026-02-17T15:21:06.377 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32585.log.gz 2026-02-17T15:21:06.377 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30092.log 2026-02-17T15:21:06.377 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23738.log 2026-02-17T15:21:06.377 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35666.log.gz 2026-02-17T15:21:06.378 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28160.log 2026-02-17T15:21:06.378 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30092.log.gz 2026-02-17T15:21:06.378 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.23738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27751.log 2026-02-17T15:21:06.378 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23738.log.gz 2026-02-17T15:21:06.379 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28160.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.28160.log.gz --verbose 2026-02-17T15:21:06.379 INFO:teuthology.orchestra.run.trial024.stderr: -- /var/log/ceph/ceph-client.admin.44739.log 2026-02-17T15:21:06.379 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40944.log 2026-02-17T15:21:06.379 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27751.log.gz 2026-02-17T15:21:06.379 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44375.log 2026-02-17T15:21:06.379 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44739.log.gz 2026-02-17T15:21:06.380 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40944.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.40944.log.gz -- 2026-02-17T15:21:06.380 INFO:teuthology.orchestra.run.trial024.stderr: /var/log/ceph/ceph-client.admin.30645.log 2026-02-17T15:21:06.380 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32617.log 2026-02-17T15:21:06.380 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44375.log: 91.9% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2026-02-17T15:21:06.380 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44375.log.gz 2026-02-17T15:21:06.380 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30645.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30645.log.gz 2026-02-17T15:21:06.380 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40610.log 2026-02-17T15:21:06.381 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32617.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32617.log.gz 2026-02-17T15:21:06.381 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21138.log 2026-02-17T15:21:06.381 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20560.log 2026-02-17T15:21:06.381 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40610.log.gz 2026-02-17T15:21:06.382 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21138.log.gz 2026-02-17T15:21:06.382 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24267.log 2026-02-17T15:21:06.382 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42513.log 2026-02-17T15:21:06.382 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20560.log.gz 2026-02-17T15:21:06.382 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40171.log 2026-02-17T15:21:06.382 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24267.log.gz 2026-02-17T15:21:06.383 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42513.log.gz 2026-02-17T15:21:06.383 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28977.log 2026-02-17T15:21:06.383 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40171.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40171.log.gz 2026-02-17T15:21:06.383 INFO:teuthology.orchestra.run.trial024.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34618.log 2026-02-17T15:21:06.383 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28977.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28977.log.gz 2026-02-17T15:21:06.384 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35198.log 2026-02-17T15:21:06.384 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45182.log 2026-02-17T15:21:06.384 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34618.log.gz 2026-02-17T15:21:06.384 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35198.log.gz 2026-02-17T15:21:06.384 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40065.log 2026-02-17T15:21:06.385 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40845.log 2026-02-17T15:21:06.385 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45182.log.gz 2026-02-17T15:21:06.385 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40065.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40065.log.gz 2026-02-17T15:21:06.385 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41518.log 2026-02-17T15:21:06.385 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30642.log 2026-02-17T15:21:06.385 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40845.log.gz 2026-02-17T15:21:06.386 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41518.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41518.log.gz 2026-02-17T15:21:06.386 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35736.log 2026-02-17T15:21:06.386 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30642.log.gz 2026-02-17T15:21:06.386 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34991.log 2026-02-17T15:21:06.386 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35736.log.gz 2026-02-17T15:21:06.386 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44179.log 2026-02-17T15:21:06.387 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30436.log 2026-02-17T15:21:06.387 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34991.log.gz 2026-02-17T15:21:06.387 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44179.log.gz 2026-02-17T15:21:06.387 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42500.log 2026-02-17T15:21:06.387 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40848.log 2026-02-17T15:21:06.388 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30436.log: /var/log/ceph/ceph-client.admin.42500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30436.log.gz 0.0% 2026-02-17T15:21:06.388 INFO:teuthology.orchestra.run.trial024.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42500.log.gz 2026-02-17T15:21:06.388 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27175.log 2026-02-17T15:21:06.388 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35766.log 2026-02-17T15:21:06.388 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40848.log.gz 2026-02-17T15:21:06.389 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41287.log 2026-02-17T15:21:06.389 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27175.log.gz 2026-02-17T15:21:06.389 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36230.log 2026-02-17T15:21:06.389 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35766.log.gz 2026-02-17T15:21:06.389 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36130.log 2026-02-17T15:21:06.389 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41287.log.gz 2026-02-17T15:21:06.390 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.36230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43283.log 2026-02-17T15:21:06.390 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36230.log.gz 2026-02-17T15:21:06.390 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.36130.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36130.log.gz 2026-02-17T15:21:06.390 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45260.log 2026-02-17T15:21:06.390 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34592.log 2026-02-17T15:21:06.391 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43283.log.gz 2026-02-17T15:21:06.391 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45260.log.gz 2026-02-17T15:21:06.391 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31867.log 2026-02-17T15:21:06.391 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34592.log.gz 2026-02-17T15:21:06.391 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31659.log 2026-02-17T15:21:06.391 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43292.log 2026-02-17T15:21:06.392 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31867.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31867.log.gz 2026-02-17T15:21:06.392 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31659.log.gz 2026-02-17T15:21:06.392 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2026-02-17T15:21:06.392 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28844.log 2026-02-17T15:21:06.392 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43292.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43292.log.gz 2026-02-17T15:21:06.392 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30202.log 2026-02-17T15:21:06.393 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28844.log.gz 2026-02-17T15:21:06.393 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41076.log 2026-02-17T15:21:06.393 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36138.log 2026-02-17T15:21:06.393 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30202.log: /var/log/ceph/ceph-client.admin.41076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30202.log.gz 2026-02-17T15:21:06.394 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31181.log 2026-02-17T15:21:06.394 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41076.log.gz 2026-02-17T15:21:06.394 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.36138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36138.log.gz 2026-02-17T15:21:06.394 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43417.log 2026-02-17T15:21:06.394 INFO:teuthology.orchestra.run.trial024.stderr: 95.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2026-02-17T15:21:06.394 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21254.log 2026-02-17T15:21:06.394 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31181.log.gz 2026-02-17T15:21:06.395 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43520.log 2026-02-17T15:21:06.395 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43417.log.gz 2026-02-17T15:21:06.395 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29408.log 2026-02-17T15:21:06.395 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21254.log.gz 2026-02-17T15:21:06.395 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41049.log 2026-02-17T15:21:06.395 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43520.log.gz 2026-02-17T15:21:06.396 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42278.log 2026-02-17T15:21:06.396 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29408.log.gz 2026-02-17T15:21:06.396 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41049.log.gz 2026-02-17T15:21:06.396 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27307.log 2026-02-17T15:21:06.397 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42278.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2026-02-17T15:21:06.397 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42278.log.gz 2026-02-17T15:21:06.397 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30773.log 2026-02-17T15:21:06.397 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27307.log.gz 2026-02-17T15:21:06.397 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35398.log 2026-02-17T15:21:06.398 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.30773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30773.log.gz 2026-02-17T15:21:06.398 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28498.log 2026-02-17T15:21:06.398 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44115.log 2026-02-17T15:21:06.398 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35398.log.gz 2026-02-17T15:21:06.398 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29616.log 2026-02-17T15:21:06.398 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28498.log.gz 2026-02-17T15:21:06.399 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29419.log 2026-02-17T15:21:06.399 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44115.log.gz 2026-02-17T15:21:06.399 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32103.log 2026-02-17T15:21:06.399 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29616.log.gz 2026-02-17T15:21:06.399 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30294.log 2026-02-17T15:21:06.400 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29419.log.gz 2026-02-17T15:21:06.400 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32103.log.gz 2026-02-17T15:21:06.400 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28301.log 2026-02-17T15:21:06.400 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30294.log.gz 2026-02-17T15:21:06.400 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26635.log 2026-02-17T15:21:06.400 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43933.log 2026-02-17T15:21:06.401 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28301.log.gz 2026-02-17T15:21:06.401 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20667.log 2026-02-17T15:21:06.401 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26635.log.gz 2026-02-17T15:21:06.401 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43933.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43933.log.gz 2026-02-17T15:21:06.401 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21873.log 2026-02-17T15:21:06.401 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20667.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2026-02-17T15:21:06.402 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20667.log.gz 2026-02-17T15:21:06.402 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20666.log 2026-02-17T15:21:06.402 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21873.log.gz 2026-02-17T15:21:06.402 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31961.log 2026-02-17T15:21:06.403 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20666.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21542.log 2026-02-17T15:21:06.403 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20666.log.gz 2026-02-17T15:21:06.403 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30879.log 2026-02-17T15:21:06.403 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31961.log.gz 2026-02-17T15:21:06.403 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21542.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.21542.log.gz --verbose 2026-02-17T15:21:06.403 INFO:teuthology.orchestra.run.trial024.stderr: -- /var/log/ceph/ceph-client.admin.23897.log 2026-02-17T15:21:06.404 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20771.log 2026-02-17T15:21:06.404 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30879.log.gz 2026-02-17T15:21:06.404 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.23897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24206.log 2026-02-17T15:21:06.404 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23897.log.gz 2026-02-17T15:21:06.404 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20771.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20771.log.gz 2026-02-17T15:21:06.404 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40368.log 2026-02-17T15:21:06.405 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21135.log 2026-02-17T15:21:06.405 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24206.log.gz 2026-02-17T15:21:06.405 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40368.log.gz 2026-02-17T15:21:06.405 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2026-02-17T15:21:06.406 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20759.log 2026-02-17T15:21:06.406 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21135.log.gz 2026-02-17T15:21:06.406 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30736.log 2026-02-17T15:21:06.406 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43592.log 2026-02-17T15:21:06.406 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20759.log.gz 2026-02-17T15:21:06.407 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32593.log 2026-02-17T15:21:06.407 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30736.log.gz 2026-02-17T15:21:06.407 INFO:teuthology.orchestra.run.trial024.stderr:gzip/var/log/ceph/ceph-client.admin.43592.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29074.log 2026-02-17T15:21:06.407 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43592.log.gz 2026-02-17T15:21:06.407 INFO:teuthology.orchestra.run.trial024.stderr: 94.3% -- replaced with /var/log/ceph/ceph.log.gz 2026-02-17T15:21:06.407 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34852.log 2026-02-17T15:21:06.407 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32593.log.gz 2026-02-17T15:21:06.408 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41934.log 2026-02-17T15:21:06.408 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29074.log.gz 2026-02-17T15:21:06.408 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43054.log 2026-02-17T15:21:06.408 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34852.log.gz 2026-02-17T15:21:06.408 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31218.log 2026-02-17T15:21:06.408 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41934.log.gz 2026-02-17T15:21:06.409 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43054.log.gz 2026-02-17T15:21:06.409 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35327.log 2026-02-17T15:21:06.409 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27964.log 2026-02-17T15:21:06.409 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31218.log.gz 2026-02-17T15:21:06.409 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45467.log 2026-02-17T15:21:06.409 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35327.log.gz 2026-02-17T15:21:06.410 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31419.log 2026-02-17T15:21:06.410 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27964.log.gz 2026-02-17T15:21:06.410 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29080.log 2026-02-17T15:21:06.410 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45467.log: /var/log/ceph/ceph-client.admin.31419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45467.log.gz 2026-02-17T15:21:06.410 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31419.log.gz 2026-02-17T15:21:06.410 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26847.log 2026-02-17T15:21:06.410 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29080.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29080.log.gz 2026-02-17T15:21:06.410 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31857.log 2026-02-17T15:21:06.411 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20665.log 2026-02-17T15:21:06.411 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26847.log.gz 2026-02-17T15:21:06.411 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31857.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31857.log.gz 2026-02-17T15:21:06.411 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40943.log 2026-02-17T15:21:06.412 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41288.log 2026-02-17T15:21:06.412 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20665.log.gz 2026-02-17T15:21:06.412 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.17009.log 2026-02-17T15:21:06.412 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40943.log.gz 2026-02-17T15:21:06.412 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41288.log.gz 2026-02-17T15:21:06.412 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36256.log 2026-02-17T15:21:06.413 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.17009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29856.log 2026-02-17T15:21:06.413 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.17009.log.gz 2026-02-17T15:21:06.413 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.36256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31216.log 2026-02-17T15:21:06.413 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36256.log.gz 2026-02-17T15:21:06.413 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29856.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29856.log.gz 2026-02-17T15:21:06.414 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24477.log 2026-02-17T15:21:06.414 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30088.log 2026-02-17T15:21:06.414 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31216.log.gz 2026-02-17T15:21:06.414 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24477.log.gz 2026-02-17T15:21:06.414 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40952.log 2026-02-17T15:21:06.415 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30088.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30088.log.gz 2026-02-17T15:21:06.415 INFO:teuthology.orchestra.run.trial024.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.24577.log 2026-02-17T15:21:06.415 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40952.log.gz 2026-02-17T15:21:06.415 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35929.log 2026-02-17T15:21:06.415 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24577.log.gz 2026-02-17T15:21:06.415 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30304.log 2026-02-17T15:21:06.416 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26834.log 2026-02-17T15:21:06.416 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35929.log.gz 2026-02-17T15:21:06.416 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30433.log 2026-02-17T15:21:06.416 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30304.log.gz 2026-02-17T15:21:06.416 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43388.log 2026-02-17T15:21:06.416 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26834.log.gz 2026-02-17T15:21:06.417 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45025.log 2026-02-17T15:21:06.417 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30433.log.gz 2026-02-17T15:21:06.417 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20561.log 2026-02-17T15:21:06.417 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43388.log.gz 2026-02-17T15:21:06.418 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29178.log 0.0% 2026-02-17T15:21:06.418 INFO:teuthology.orchestra.run.trial024.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45025.log.gz 2026-02-17T15:21:06.418 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45130.log 2026-02-17T15:21:06.418 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20561.log.gz 2026-02-17T15:21:06.418 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29178.log.gz 2026-02-17T15:21:06.418 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40068.log 2026-02-17T15:21:06.418 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27627.log 2026-02-17T15:21:06.419 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45130.log.gz 2026-02-17T15:21:06.419 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40068.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40068.log.gz 2026-02-17T15:21:06.419 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23906.log 2026-02-17T15:21:06.419 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.27627.log: -- /var/log/ceph/ceph-client.admin.28940.log 2026-02-17T15:21:06.419 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27627.log.gz 2026-02-17T15:21:06.419 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.23906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23906.log.gz 2026-02-17T15:21:06.419 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16622.log 2026-02-17T15:21:06.420 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28940.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28940.log.gz 2026-02-17T15:21:06.420 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40030.log 2026-02-17T15:21:06.420 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.16622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44272.log 2026-02-17T15:21:06.420 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16622.log.gz 2026-02-17T15:21:06.420 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40030.log.gz 2026-02-17T15:21:06.420 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40509.log 2026-02-17T15:21:06.421 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35866.log 2026-02-17T15:21:06.421 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44272.log.gz 2026-02-17T15:21:06.421 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40509.log.gz 2026-02-17T15:21:06.421 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21337.log 2026-02-17T15:21:06.421 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31219.log 2026-02-17T15:21:06.421 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35866.log: /var/log/ceph/ceph-client.admin.21337.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21337.log.gz 2026-02-17T15:21:06.422 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35866.log.gz 2026-02-17T15:21:06.422 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28406.log 2026-02-17T15:21:06.422 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28634.log 2026-02-17T15:21:06.422 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31219.log.gz 2026-02-17T15:21:06.422 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21824.log 2026-02-17T15:21:06.422 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28406.log.gz 2026-02-17T15:21:06.423 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29964.log 2026-02-17T15:21:06.423 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28634.log.gz 2026-02-17T15:21:06.423 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21776.log 2026-02-17T15:21:06.423 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21824.log.gz 2026-02-17T15:21:06.423 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21234.log 2026-02-17T15:21:06.423 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29964.log.gz 2026-02-17T15:21:06.424 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21776.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21776.log.gz 2026-02-17T15:21:06.424 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30983.log 2026-02-17T15:21:06.424 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21234.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.21234.log.gz 2026-02-17T15:21:06.424 INFO:teuthology.orchestra.run.trial024.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44869.log 2026-02-17T15:21:06.424 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27183.log 2026-02-17T15:21:06.425 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30983.log.gz 2026-02-17T15:21:06.425 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.44869.log: -- /var/log/ceph/ceph-client.admin.16867.log 2026-02-17T15:21:06.425 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44869.log.gz 2026-02-17T15:21:06.425 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27183.log.gz 2026-02-17T15:21:06.425 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42173.log 2026-02-17T15:21:06.425 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42405.log 2026-02-17T15:21:06.426 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.16867.log: /var/log/ceph/ceph-client.admin.42173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16867.log.gz 2026-02-17T15:21:06.426 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42173.log.gz 2026-02-17T15:21:06.426 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42276.log 2026-02-17T15:21:06.426 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.42405.log: -- /var/log/ceph/ceph-client.admin.27618.log 2026-02-17T15:21:06.426 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42405.log.gz 2026-02-17T15:21:06.427 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29965.log 2026-02-17T15:21:06.427 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42276.log.gz 2026-02-17T15:21:06.427 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29853.log 2026-02-17T15:21:06.427 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27618.log.gz 2026-02-17T15:21:06.427 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29965.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.29965.log.gz /var/log/ceph/ceph-client.admin.29182.log 2026-02-17T15:21:06.427 INFO:teuthology.orchestra.run.trial024.stderr: 2026-02-17T15:21:06.428 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29853.log.gz 2026-02-17T15:21:06.428 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42608.log 2026-02-17T15:21:06.428 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40951.log 2026-02-17T15:21:06.428 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29182.log.gz 2026-02-17T15:21:06.428 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24374.log 2026-02-17T15:21:06.428 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42608.log.gz 2026-02-17T15:21:06.428 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40951.log.gz 2026-02-17T15:21:06.429 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44947.log 2026-02-17T15:21:06.429 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42266.log 2026-02-17T15:21:06.429 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24374.log.gz 2026-02-17T15:21:06.429 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42636.log 2026-02-17T15:21:06.430 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44947.log: /var/log/ceph/ceph-client.admin.42266.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42266.log.gz 2026-02-17T15:21:06.430 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44947.log.gz 2026-02-17T15:21:06.430 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30201.log 2026-02-17T15:21:06.430 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42636.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.42636.log.gz -- 2026-02-17T15:21:06.430 INFO:teuthology.orchestra.run.trial024.stderr: /var/log/ceph/ceph-client.admin.40160.log 2026-02-17T15:21:06.430 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30985.log 2026-02-17T15:21:06.430 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30201.log.gz 2026-02-17T15:21:06.431 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40160.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40160.log.gz 2026-02-17T15:21:06.431 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36029.log 2026-02-17T15:21:06.431 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30985.log.gz 2026-02-17T15:21:06.431 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29227.log 2026-02-17T15:21:06.431 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.36029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28173.log 2026-02-17T15:21:06.431 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36029.log.gz 2026-02-17T15:21:06.432 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16843.log 2026-02-17T15:21:06.432 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29227.log.gz 2026-02-17T15:21:06.432 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29420.log 2026-02-17T15:21:06.432 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28173.log.gz 2026-02-17T15:21:06.432 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.16843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16843.log.gz 2026-02-17T15:21:06.432 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31660.log 2026-02-17T15:21:06.433 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29420.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.29311.log -- replaced with /var/log/ceph/ceph-client.admin.29420.log.gz 2026-02-17T15:21:06.433 INFO:teuthology.orchestra.run.trial024.stderr: 2026-02-17T15:21:06.433 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31660.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31660.log.gz 2026-02-17T15:21:06.433 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44791.log 2026-02-17T15:21:06.433 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21231.log 2026-02-17T15:21:06.433 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29311.log: /var/log/ceph/ceph-client.admin.44791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29311.log.gz 2026-02-17T15:21:06.433 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44791.log.gz 2026-02-17T15:21:06.434 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35435.log 2026-02-17T15:21:06.434 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41732.log 2026-02-17T15:21:06.434 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21231.log.gz 2026-02-17T15:21:06.434 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35435.log.gz 2026-02-17T15:21:06.434 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26843.log 2026-02-17T15:21:06.435 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28502.log 2026-02-17T15:21:06.435 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41732.log.gz 2026-02-17T15:21:06.435 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20446.log 2026-02-17T15:21:06.435 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26843.log.gz 2026-02-17T15:21:06.435 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45390.log 2026-02-17T15:21:06.435 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28502.log.gz 2026-02-17T15:21:06.436 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20446.log.gz 2026-02-17T15:21:06.436 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29621.log 2026-02-17T15:21:06.436 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24110.log 2026-02-17T15:21:06.436 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45390.log.gz 2026-02-17T15:21:06.436 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29621.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29621.log.gz 2026-02-17T15:21:06.436 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41590.log 2026-02-17T15:21:06.436 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44141.log 2026-02-17T15:21:06.437 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24110.log.gz 2026-02-17T15:21:06.437 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41590.log.gz 2026-02-17T15:21:06.437 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31658.log 2026-02-17T15:21:06.437 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30164.log 2026-02-17T15:21:06.437 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44141.log.gz 2026-02-17T15:21:06.438 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31658.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.31658.log.gz 2026-02-17T15:21:06.438 INFO:teuthology.orchestra.run.trial024.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.28742.log 2026-02-17T15:21:06.438 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42849.log 2026-02-17T15:21:06.438 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30164.log.gz 2026-02-17T15:21:06.438 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28742.log.gz 2026-02-17T15:21:06.438 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27858.log 2026-02-17T15:21:06.439 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43294.log 2026-02-17T15:21:06.439 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42849.log.gz 2026-02-17T15:21:06.439 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20863.log 2026-02-17T15:21:06.439 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27858.log.gz 2026-02-17T15:21:06.439 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43294.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43294.log.gz 2026-02-17T15:21:06.439 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36028.log 2026-02-17T15:21:06.440 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41077.log 2026-02-17T15:21:06.440 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20863.log.gz 2026-02-17T15:21:06.440 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.36028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27625.log 2026-02-17T15:21:06.440 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36028.log.gz 2026-02-17T15:21:06.441 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41077.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41077.log.gz 2026-02-17T15:21:06.441 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32207.log 2026-02-17T15:21:06.441 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30305.log 2026-02-17T15:21:06.441 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27625.log.gz 2026-02-17T15:21:06.441 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32207.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2026-02-17T15:21:06.441 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32207.log.gz 2026-02-17T15:21:06.442 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30305.log.gz 2026-02-17T15:21:06.442 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20563.log 2026-02-17T15:21:06.442 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29864.log 2026-02-17T15:21:06.443 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23800.log 2026-02-17T15:21:06.443 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20563.log.gz 2026-02-17T15:21:06.443 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41618.log 2026-02-17T15:21:06.443 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29864.log.gz 2026-02-17T15:21:06.443 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.23800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35436.log 2026-02-17T15:21:06.443 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23800.log.gz 2026-02-17T15:21:06.444 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42370.log 2026-02-17T15:21:06.444 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41618.log.gz 2026-02-17T15:21:06.444 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42511.log 2026-02-17T15:21:06.444 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35436.log.gz 2026-02-17T15:21:06.444 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42370.log.gz 2026-02-17T15:21:06.444 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32592.log 2026-02-17T15:21:06.445 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30435.log 2026-02-17T15:21:06.445 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42511.log.gz 2026-02-17T15:21:06.445 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32592.log.gz 2026-02-17T15:21:06.445 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21850.log 2026-02-17T15:21:06.445 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30435.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30435.log.gz 2026-02-17T15:21:06.445 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35088.log 2026-02-17T15:21:06.446 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21850.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21850.log.gz 2026-02-17T15:21:06.446 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40612.log 2026-02-17T15:21:06.446 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28529.log 2026-02-17T15:21:06.446 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35088.log.gz 2026-02-17T15:21:06.447 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32689.log 2026-02-17T15:21:06.447 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40612.log.gz 2026-02-17T15:21:06.447 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44635.log 2026-02-17T15:21:06.447 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28529.log.gz 2026-02-17T15:21:06.447 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30774.log 2026-02-17T15:21:06.447 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32689.log.gz 2026-02-17T15:21:06.448 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30877.log 2026-02-17T15:21:06.448 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44635.log.gz 2026-02-17T15:21:06.448 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34653.log 2026-02-17T15:21:06.448 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30774.log.gz 2026-02-17T15:21:06.448 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31109.log 2026-02-17T15:21:06.448 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30877.log.gz 2026-02-17T15:21:06.449 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21244.log 2026-02-17T15:21:06.449 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34653.log.gz 2026-02-17T15:21:06.449 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32741.log 2026-02-17T15:21:06.449 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31109.log.gz 2026-02-17T15:21:06.450 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21643.log 2026-02-17T15:21:06.450 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21244.log.gz 2026-02-17T15:21:06.450 INFO:teuthology.orchestra.run.trial024.stderr:gzip/var/log/ceph/ceph-client.admin.32741.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.31467.log 2026-02-17T15:21:06.450 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32741.log.gz 2026-02-17T15:21:06.450 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21643.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.21643.log.gz -- 2026-02-17T15:21:06.450 INFO:teuthology.orchestra.run.trial024.stderr: /var/log/ceph/ceph-client.admin.21620.log 2026-02-17T15:21:06.451 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44713.log 2026-02-17T15:21:06.451 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31467.log.gz 2026-02-17T15:21:06.451 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34994.log 2026-02-17T15:21:06.451 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21620.log.gz 2026-02-17T15:21:06.451 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44713.log.gz 2026-02-17T15:21:06.451 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30537.log 2026-02-17T15:21:06.452 INFO:teuthology.orchestra.run.trial024.stderr:gzip/var/log/ceph/ceph-client.admin.34994.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27860.log 2026-02-17T15:21:06.452 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34994.log.gz 2026-02-17T15:21:06.452 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30537.log.gz 2026-02-17T15:21:06.452 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44843.log 2026-02-17T15:21:06.453 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20525.log 2026-02-17T15:21:06.453 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27860.log.gz 2026-02-17T15:21:06.453 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44843.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43735.log 2026-02-17T15:21:06.453 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44843.log.gz 2026-02-17T15:21:06.453 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21898.log 2026-02-17T15:21:06.453 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20525.log.gz 2026-02-17T15:21:06.454 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40712.log 2026-02-17T15:21:06.454 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43735.log.gz 2026-02-17T15:21:06.454 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21898.log.gz 2026-02-17T15:21:06.454 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20499.log 2026-02-17T15:21:06.454 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27184.log 2026-02-17T15:21:06.454 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40712.log.gz 2026-02-17T15:21:06.455 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28069.log 2026-02-17T15:21:06.455 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20499.log.gz 2026-02-17T15:21:06.455 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27184.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27184.log.gz 2026-02-17T15:21:06.455 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40403.log 2026-02-17T15:21:06.456 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35433.log 2026-02-17T15:21:06.456 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28069.log.gz 2026-02-17T15:21:06.456 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.40403.log: /var/log/ceph/ceph-client.admin.43055.log 2026-02-17T15:21:06.456 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40403.log.gz 2026-02-17T15:21:06.456 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35433.log.gz 2026-02-17T15:21:06.456 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29421.log 2026-02-17T15:21:06.457 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21137.log 2026-02-17T15:21:06.457 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43055.log.gz 2026-02-17T15:21:06.457 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27617.log 2026-02-17T15:21:06.457 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29421.log.gz 2026-02-17T15:21:06.457 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21345.log 2026-02-17T15:21:06.458 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21137.log.gz 2026-02-17T15:21:06.458 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21136.log 2026-02-17T15:21:06.458 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27617.log.gz 2026-02-17T15:21:06.458 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31315.log 2026-02-17T15:21:06.458 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21345.log.gz 2026-02-17T15:21:06.459 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43629.log 2026-02-17T15:21:06.459 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21136.log.gz 2026-02-17T15:21:06.459 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29073.log 2026-02-17T15:21:06.459 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31315.log.gz 2026-02-17T15:21:06.459 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41622.log 2026-02-17T15:21:06.459 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43629.log.gz 2026-02-17T15:21:06.460 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29073.log.gz 2026-02-17T15:21:06.460 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41386.log 2026-02-17T15:21:06.460 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31525.log 2026-02-17T15:21:06.460 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41622.log.gz 2026-02-17T15:21:06.460 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43492.log 2026-02-17T15:21:06.460 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41386.log.gz 2026-02-17T15:21:06.461 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44921.log 2026-02-17T15:21:06.461 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31525.log.gz 2026-02-17T15:21:06.461 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35541.log 2026-02-17T15:21:06.461 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43492.log.gz 2026-02-17T15:21:06.462 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20915.log 2026-02-17T15:21:06.462 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44921.log.gz 2026-02-17T15:21:06.462 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31763.log 2026-02-17T15:21:06.462 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35541.log.gz 2026-02-17T15:21:06.462 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20915.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20915.log.gz -5 2026-02-17T15:21:06.462 INFO:teuthology.orchestra.run.trial024.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38130.log 2026-02-17T15:21:06.463 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42950.log 2026-02-17T15:21:06.463 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31763.log.gz 2026-02-17T15:21:06.463 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.38130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44895.log 2026-02-17T15:21:06.463 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38130.log.gz 2026-02-17T15:21:06.463 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42950.log.gz 2026-02-17T15:21:06.463 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36137.log 2026-02-17T15:21:06.464 INFO:teuthology.orchestra.run.trial024.stderr:gzip/var/log/ceph/ceph-client.admin.44895.log: -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2026-02-17T15:21:06.464 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44895.log.gz 2026-02-17T15:21:06.464 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.36137.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36137.log.gz 2026-02-17T15:21:06.464 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41961.log 2026-02-17T15:21:06.465 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34773.log 2026-02-17T15:21:06.465 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24102.log 2026-02-17T15:21:06.465 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41961.log.gz 2026-02-17T15:21:06.465 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34773.log.gz 2026-02-17T15:21:06.465 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32715.log 2026-02-17T15:21:06.466 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24102.log.gz 2026-02-17T15:21:06.466 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43841.log 2026-02-17T15:21:06.466 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30984.log 2026-02-17T15:21:06.466 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32715.log.gz 2026-02-17T15:21:06.466 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.43841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45078.log 2026-02-17T15:21:06.466 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43841.log.gz 2026-02-17T15:21:06.467 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30984.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30984.log.gz 2026-02-17T15:21:06.467 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31323.log 2026-02-17T15:21:06.467 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.45078.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45078.log.gz 2026-02-17T15:21:06.467 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35303.log 2026-02-17T15:21:06.468 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34752.log 2026-02-17T15:21:06.468 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31323.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31323.log.gz 2026-02-17T15:21:06.468 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35303.log.gz 2026-02-17T15:21:06.468 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26638.log 2026-02-17T15:21:06.468 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.34752.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30632.log 2026-02-17T15:21:06.468 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34752.log.gz 2026-02-17T15:21:06.469 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.26638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26638.log.gz 2026-02-17T15:21:06.469 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40404.log 2026-02-17T15:21:06.469 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.30632.log: -- /var/log/ceph/ceph-client.admin.28706.log 2026-02-17T15:21:06.469 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30632.log.gz 2026-02-17T15:21:06.469 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27522.log 2026-02-17T15:21:06.469 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40404.log.gz 2026-02-17T15:21:06.469 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28706.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28706.log.gz 2026-02-17T15:21:06.470 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28978.log 2026-02-17T15:21:06.470 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32386.log 2026-02-17T15:21:06.470 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27522.log.gz 2026-02-17T15:21:06.470 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24373.log 2026-02-17T15:21:06.470 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28978.log.gz 2026-02-17T15:21:06.470 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32386.log.gz 2026-02-17T15:21:06.471 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21594.log 2026-02-17T15:21:06.471 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27079.log 2026-02-17T15:21:06.471 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24373.log.gz 2026-02-17T15:21:06.471 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.21594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21594.log.gz 2026-02-17T15:21:06.471 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.16766.log 2026-02-17T15:21:06.472 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.27079.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27079.log.gz -5 2026-02-17T15:21:06.472 INFO:teuthology.orchestra.run.trial024.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41495.log 2026-02-17T15:21:06.472 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.16766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31623.log 2026-02-17T15:21:06.472 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.16766.log.gz 2026-02-17T15:21:06.473 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.41495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41495.log.gz 2026-02-17T15:21:06.473 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40004.log 2026-02-17T15:21:06.473 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28297.log 2026-02-17T15:21:06.473 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31623.log.gz 2026-02-17T15:21:06.473 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42745.log 2026-02-17T15:21:06.473 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40004.log.gz 2026-02-17T15:21:06.474 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28976.log 2026-02-17T15:21:06.474 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28297.log.gz 2026-02-17T15:21:06.474 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.42745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42745.log.gz 2026-02-17T15:21:06.474 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29720.log 2026-02-17T15:21:06.474 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.28976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28976.log.gz 2026-02-17T15:21:06.474 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29546.log 2026-02-17T15:21:06.475 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40066.log 2026-02-17T15:21:06.475 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29720.log: 0.0%/var/log/ceph/ceph-client.admin.29546.log: -- replaced with /var/log/ceph/ceph-client.admin.29720.log.gz 2026-02-17T15:21:06.475 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29546.log.gz 2026-02-17T15:21:06.475 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36030.log 2026-02-17T15:21:06.475 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40066.log.gz 2026-02-17T15:21:06.475 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40264.log 2026-02-17T15:21:06.476 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44479.log 2026-02-17T15:21:06.476 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.36030.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36030.log.gz 2026-02-17T15:21:06.476 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40264.log.gz 2026-02-17T15:21:06.476 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41252.log 2026-02-17T15:21:06.476 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24756.log 2026-02-17T15:21:06.477 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44479.log: /var/log/ceph/ceph-client.admin.41252.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41252.log.gz 2026-02-17T15:21:06.477 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44479.log.gz 2026-02-17T15:21:06.477 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35094.log 2026-02-17T15:21:06.477 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.24756.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.24756.log.gz /var/log/ceph/ceph-client.admin.44275.log 2026-02-17T15:21:06.477 INFO:teuthology.orchestra.run.trial024.stderr: 2026-02-17T15:21:06.477 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35094.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35094.log.gz 2026-02-17T15:21:06.477 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31519.log 2026-02-17T15:21:06.478 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.44275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29315.log 2026-02-17T15:21:06.478 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44275.log.gz 2026-02-17T15:21:06.478 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.31519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31519.log.gz 2026-02-17T15:21:06.478 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.20993.log 2026-02-17T15:21:06.478 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.29315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29315.log.gz 2026-02-17T15:21:06.478 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32100.log 2026-02-17T15:21:06.479 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40843.log 2026-02-17T15:21:06.479 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.20993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.20993.log.gz 2026-02-17T15:21:06.479 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30538.log 2026-02-17T15:21:06.479 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32100.log.gz 2026-02-17T15:21:06.479 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.40843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40843.log.gz 2026-02-17T15:21:06.479 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32389.log 2026-02-17T15:21:06.480 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.30538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35636.log 2026-02-17T15:21:06.480 INFO:teuthology.orchestra.run.trial024.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30538.log.gz 2026-02-17T15:21:06.480 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.32389.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32389.log.gz 2026-02-17T15:21:06.480 INFO:teuthology.orchestra.run.trial024.stderr:/var/log/ceph/ceph-client.admin.35636.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35636.log.gz 2026-02-17T15:21:06.493 INFO:teuthology.orchestra.run.trial024.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2026-02-17T15:21:06.533 INFO:teuthology.orchestra.run.trial024.stderr: 90.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2026-02-17T15:21:19.707 INFO:teuthology.orchestra.run.trial024.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2026-02-17T15:21:20.373 INFO:teuthology.orchestra.run.trial024.stderr: 95.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2026-02-17T15:21:38.913 INFO:teuthology.orchestra.run.trial024.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2026-02-17T15:23:00.661 INFO:teuthology.orchestra.run.trial024.stderr: 91.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2026-02-17T15:23:00.663 INFO:teuthology.orchestra.run.trial024.stderr: 2026-02-17T15:23:00.663 INFO:teuthology.orchestra.run.trial024.stderr:real 1m54.409s 2026-02-17T15:23:00.663 INFO:teuthology.orchestra.run.trial024.stderr:user 2m47.868s 2026-02-17T15:23:00.663 INFO:teuthology.orchestra.run.trial024.stderr:sys 0m7.178s 2026-02-17T15:23:00.664 INFO:tasks.ceph:Archiving logs... 2026-02-17T15:23:00.664 DEBUG:teuthology.misc:Transferring archived files from trial024:/var/log/ceph to /home/teuthworker/mnt/teuthology/skanta-2026-02-17_04:10:36-rados-wip-bharath20-testing-2026-02-13-0906-squid-distro-default-trial/51568/remote/trial024/log 2026-02-17T15:23:00.666 DEBUG:teuthology.orchestra.run.trial024:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-17T15:23:14.925 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-17T15:23:14.937 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-17T15:23:14.937 DEBUG:teuthology.orchestra.run.trial024:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-17T15:23:15.011 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-17T15:23:15.022 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-17T15:23:15.023 DEBUG:teuthology.orchestra.run.trial024:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-17T15:23:15.335 INFO:teuthology.orchestra.run.trial024.stdout: remote refid st t when poll reach delay offset jitter 2026-02-17T15:23:15.335 INFO:teuthology.orchestra.run.trial024.stdout:============================================================================== 2026-02-17T15:23:15.335 INFO:teuthology.orchestra.run.trial024.stdout:*ntp2.zeus.frume 209.180.247.49 2 u 31 64 377 29.777 +3.117 1.223 2026-02-17T15:23:15.335 INFO:teuthology.orchestra.run.trial024.stdout:-LAX.CALTICK.NET 17.253.83.251 2 u 28 64 377 62.925 +1.696 1.325 2026-02-17T15:23:15.335 INFO:teuthology.orchestra.run.trial024.stdout:+73.185.182.209 204.117.185.216 2 u 29 64 377 39.708 +0.767 3.152 2026-02-17T15:23:15.335 INFO:teuthology.orchestra.run.trial024.stdout:+ntp.maxhost.io 132.163.97.4 2 u 36 64 377 69.671 -0.220 4.986 2026-02-17T15:23:15.336 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-17T15:23:15.347 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-17T15:23:15.348 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-17T15:23:15.363 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-17T15:23:15.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-17T15:23:15.391 INFO:teuthology.task.internal:Duration was 777.259205 seconds 2026-02-17T15:23:15.392 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-17T15:23:15.407 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-17T15:23:15.407 DEBUG:teuthology.orchestra.run.trial024:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-17T15:23:15.433 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-17T15:23:15.433 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial024.front.sepia.ceph.com 2026-02-17T15:23:15.433 DEBUG:teuthology.orchestra.run.trial024:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-17T15:23:15.482 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-17T15:23:15.482 DEBUG:teuthology.orchestra.run.trial024:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-17T15:23:15.688 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-17T15:23:15.688 DEBUG:teuthology.orchestra.run.trial024:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-17T15:23:15.698 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-17T15:23:15.699 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-17T15:23:15.699 INFO:teuthology.orchestra.run.trial024.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-17T15:23:15.699 INFO:teuthology.orchestra.run.trial024.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-17T15:23:15.699 INFO:teuthology.orchestra.run.trial024.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-17T15:23:15.745 INFO:teuthology.orchestra.run.trial024.stderr: 82.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-17T15:23:15.747 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-17T15:23:15.763 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-17T15:23:15.764 DEBUG:teuthology.orchestra.run.trial024:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-17T15:23:15.796 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-17T15:23:15.812 DEBUG:teuthology.orchestra.run.trial024:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-17T15:23:15.847 INFO:teuthology.orchestra.run.trial024.stdout:kernel.core_pattern = core 2026-02-17T15:23:15.856 DEBUG:teuthology.orchestra.run.trial024:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-17T15:23:15.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-17T15:23:15.903 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-17T15:23:15.919 INFO:teuthology.task.internal:Transferring archived files... 2026-02-17T15:23:15.920 DEBUG:teuthology.misc:Transferring archived files from trial024:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/skanta-2026-02-17_04:10:36-rados-wip-bharath20-testing-2026-02-13-0906-squid-distro-default-trial/51568/remote/trial024 2026-02-17T15:23:15.920 DEBUG:teuthology.orchestra.run.trial024:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-17T15:23:15.973 INFO:teuthology.task.internal:Removing archive directory... 2026-02-17T15:23:15.973 DEBUG:teuthology.orchestra.run.trial024:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-17T15:23:16.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-17T15:23:16.025 INFO:teuthology.task.internal:Not uploading archives. 2026-02-17T15:23:16.025 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-17T15:23:16.040 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-17T15:23:16.040 DEBUG:teuthology.orchestra.run.trial024:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-17T15:23:16.055 INFO:teuthology.orchestra.run.trial024.stdout: 1180220 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 17 15:23 /home/ubuntu/cephtest 2026-02-17T15:23:16.057 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-17T15:23:16.074 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-17T15:23:16.110 INFO:teuthology.run:Summary data: description: rados/singleton/{all/ec-inconsistent-hinfo mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zstd rados supported-random-distro$/{ubuntu_latest}} duration: 777.259204864502 flavor: default owner: scheduled_skanta@soko04.front.sepia.ceph.com success: true 2026-02-17T15:23:16.110 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-17T15:23:16.226 INFO:teuthology.run:pass